Skip to content

POC: Try to optimize columnized_output_exprs - #13018

Closed
goldmedal wants to merge 4 commits into
apache:mainfrom
goldmedal:poc/opt-plan-many-col-agg
Closed

goldmedal wants to merge 4 commits into
apache:mainfrom
goldmedal:poc/opt-plan-many-col-agg

Conversation

@goldmedal

@goldmedal goldmedal commented Oct 19, 2024 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Just a POC for #13015 (comment)

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@github-actions github-actions Bot added logical-expr Logical plan and expressions common Related to common crate labels Oct 19, 2024
Comment thread datafusion/common/src/dfschema.rs Outdated
.map(|(qualifier, field)| {
ColumnReference::new(qualifier.cloned(), field.name().as_str())
})
.collect()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This Vec can be avoided by returning iterator?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, nice idea. Thanks!

@goldmedal

Copy link
Copy Markdown
Contributor Author

The latest commit doesn't make the performance better. Just clean up the codebase.

@alamb

alamb commented Oct 24, 2024 •

Copy link
Copy Markdown
Contributor

I may be misreading this, but it seems like columnize_output_expr is called in a loop for each expression being projected:

for e in expr {
let e = e.into();
match e {
Expr::Wildcard { .. } => projected_expr.push(e),
_ => projected_expr.push(columnize_expr(normalize_col(e, &plan)?, &plan)?),
}
}

However, then for each expr it builds up the same hash table:

let output_exprs = match input.columnized_output_exprs() {
Ok(exprs) if !exprs.is_empty() => exprs,
_ => return Ok(e),
};
let exprs_map: HashMap<&Expr, Column> = output_exprs.into_iter().collect();

Maybe we could refactor the code so the hash table is built once rather than once per expression? Something like

let columizer = Columnizer(plan);
for e in exprs {
  columnizer.columnize(e)
}

🤔

coupled with your changes to avoid hash map might make a significant difference in planning

@goldmedal

Copy link
Copy Markdown
Contributor Author

coupled with your changes to avoid hash map might make a significant difference in planning

Thanks, @alamb. I'll check it in a few days. If it makes sense, I will do the benchmark again 👍

@goldmedal

Copy link
Copy Markdown
Contributor Author

However, then for each expr it builds up the same hash table:

let output_exprs = match input.columnized_output_exprs() {
Ok(exprs) if !exprs.is_empty() => exprs,
_ => return Ok(e),
};
let exprs_map: HashMap<&Expr, Column> = output_exprs.into_iter().collect();

It doesn't work for this case because the hash tables are different. The case is

SELECT max(a1), max(a2), ... max(a200) from t

I think sharing the hash map may work in some cases like

SELECT max(a1), max(a1) + 1, max(a1) +2, ... from t

We only need to hash max(a1) once, and then look up the column reference many times. In physical_select_aggregates_from_200 case, we still need to hash every expression.

@alamb

alamb commented Oct 28, 2024

Copy link
Copy Markdown
Contributor

Thanks for double checking @goldmedal

@github-actions

Copy link
Copy Markdown

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

@github-actions github-actions Bot added the Stale PR has not had any activity for some time label Dec 28, 2024
@goldmedal goldmedal closed this Dec 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common Related to common crate logical-expr Logical plan and expressions Stale PR has not had any activity for some time

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants