Repository navigation
Conversation
| .map(|(qualifier, field)| { | ||
| ColumnReference::new(qualifier.cloned(), field.name().as_str()) | ||
| }) | ||
| .collect() |
There was a problem hiding this comment.
This Vec can be avoided by returning iterator?
There was a problem hiding this comment.
Indeed, nice idea. Thanks!
|
The latest commit doesn't make the performance better. Just clean up the codebase. |
|
I may be misreading this, but it seems like columnize_output_expr is called in a loop for each expression being projected: datafusion/datafusion/expr/src/logical_plan/builder.rs Lines 1505 to 1511 in ac827ab However, then for each expr it builds up the same hash table: datafusion/datafusion/expr/src/utils.rs Lines 879 to 883 in ac827ab Maybe we could refactor the code so the hash table is built once rather than once per expression? Something like let columizer = Columnizer(plan);
for e in exprs {
columnizer.columnize(e)
}🤔 coupled with your changes to avoid hash map might make a significant difference in planning |
Thanks, @alamb. I'll check it in a few days. If it makes sense, I will do the benchmark again 👍 |
It doesn't work for this case because the hash tables are different. The case is I think sharing the hash map may work in some cases like We only need to hash |
|
Thanks for double checking @goldmedal |
|
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days. |
Which issue does this PR close?
Just a POC for #13015 (comment)
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?