You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Consider introducing unique expression IDs in Logical/Physical plan #8379
/** * A globally unique id for a given named expression. * Used to identify which attribute output by a relation is being * referenced in a subsequent computation. * * The `id` field is unique within a given JVM, while the `uuid` is used to uniquely identify JVMs.*/caseclassExprId(id: Long, jvmId: UUID) {
overridedefequals(other: Any):Boolean= other match {
caseExprId(id, jvmId) =>this.id == id &&this.jvmId == jvmId
case _ =>false
}
overridedefhashCode():Int= id.hashCode()
}
Is it worth as attempting to introduce something similar in DataFusion?
There are issues being caused by rules in the optimizer comparing directly on column name leading to bugs when duplicate names appear, such as #8374
If during the analysis of a plan we can assign unique numeric IDs for columns, we could check for column equality based on these IDs and not need to compare string names.
The obvious downside would be this seems like a large effort in refactoring, not to mention breaking changes.
Describe the solution you'd like
Consider introduction of unique ID for columns/expressions to potentially simplify optimization/planning code
Describe alternatives you've considered
Don't do this (large refactoring effort? breaking changes?)
Additional context
Just a thought I had bouncing in my head, would appreciate to hear more thoughts on this (even if this seems unfeasible), or if there was already some prior discussion on a similar topic
Actually I think I was off the mark on what ExprId is intended to do, it seems it would be more useful if there were a new LogicalExpr enum such as AttributeReference, which would refer to an expr from the parent plan by ExprId
Like given a logical plan:
Projection: a.int_col, b.double_col, CAST(a.date_string_col AS Utf8)
Inner Join: a.int_col = b.int_col
SubqueryAlias: a
Projection: alltypes_plain.int_col, alltypes_plain.date_string_col
Filter: alltypes_plain.id > Int32(1)
TableScan: alltypes_plain projection=[id, int_col, date_string_col], partial_filters=[alltypes_plain.id > Int32(1)]
SubqueryAlias: b
Projection: alltypes_plain.int_col, alltypes_plain.double_col
Filter: CAST(alltypes_plain.tinyint_col AS Float64) < alltypes_plain.double_col
TableScan: alltypes_plain projection=[tinyint_col, int_col, double_col], partial_filters=[CAST(alltypes_plain.tinyint_col AS Float64) < alltypes_plain.double_col]
That top level projection has a.int_col as a Column for example, which when turned into physical plan needs to search the parent schema by name
Whereas with exprid's, it could be possible for a.int_col to be an AttributeReference which references the parent expr list to point to which expr it references by id.
And I think each new expr would have a new ID.
Honestly I could be way off the mark here on the usages/benefits of exprid 😅
It's just something I was thinking about, especially in relation to how verbose it can be to check if columns are the same when taking into account table, schema and catalog parts of the identifier for a column
So instead of having to find the original column of a projected column in a logical plan via name during logical optimization/physical planning, could have that done once off in an analyzer rule pass then afterwards use exprids
What you describe seems similar to the usizeindex in physical_planColumn references. And as I recall @houqp added exactly to handle this case of ambiguous schema 🤔
I believe Postgres has a similar way of addressing fields (it does so by index rather than column name) in its logical exprs.
The downside I remember from working wit postgres was that there were many potential hard to track down issues when the indexes got messed up.
It might be interesting to see what this would look like in DataFusion 🤔
Related to this, SQLite and Postgres allow duplicate column names in results. From SQLite tests:
SELECT+COUNT( * ) AS col1, -92-+-47AS col1
Datafusion errors out with
query failed: Error during planning: Projections require unique expression names but the expression "COUNT(*) AS col1" at position 0 and "Int64(-92) - Int64(-47) AS col1" at position 1 have the same name. Consider aliasing ("AS") one of them.
Maybe after/as part of implementing this, Datafusion should relax that restriction?
Maybe after/as part of implementing this, Datafusion should relax that restriction?
One challenge is that most of the arrow-rs functionality in RecordBatch, for example, assumes unique column names (find_by_name for example). I don't think the Arrow spec itself restricts names to be unique, but I think the assumption may run pretty deep in the implementations
Activity
alamb commentedon Dec 1, 2023
I wonder what "unique" means ? Like every newly created
Exprgets some sort of id?Some examples:
Jefffrey commentedon Dec 1, 2023
Actually I think I was off the mark on what
ExprIdis intended to do, it seems it would be more useful if there were a new LogicalExpr enum such asAttributeReference, which would refer to an expr from the parent plan by ExprIdLike given a logical plan:
That top level projection has
a.int_colas aColumnfor example, which when turned into physical plan needs to search the parent schema by namehttps://github.com/apache/arrow-datafusion/blob/a6e6d3fab083839239ef81cf3a3546dd8929a541/datafusion/core/src/physical_planner.rs#L879-L891
Whereas with exprid's, it could be possible for
a.int_colto be an AttributeReference which references the parent expr list to point to which expr it references by id.And I think each new expr would have a new ID.
Honestly I could be way off the mark here on the usages/benefits of exprid 😅
It's just something I was thinking about, especially in relation to how verbose it can be to check if columns are the same when taking into account table, schema and catalog parts of the identifier for a column
So instead of having to find the original column of a projected column in a logical plan via name during logical optimization/physical planning, could have that done once off in an analyzer rule pass then afterwards use exprids
alamb commentedon Dec 3, 2023
What you describe seems similar to the
usizeindexinphysical_planColumn references. And as I recall @houqp added exactly to handle this case of ambiguous schema 🤔I believe Postgres has a similar way of addressing fields (it does so by index rather than column name) in its logical exprs.
The downside I remember from working wit postgres was that there were many potential hard to track down issues when the indexes got messed up.
It might be interesting to see what this would look like in DataFusion 🤔
tv42 commentedon Dec 8, 2023
Related to this, SQLite and Postgres allow duplicate column names in results. From SQLite tests:
Datafusion errors out with
Maybe after/as part of implementing this, Datafusion should relax that restriction?
alamb commentedon Dec 9, 2023
One challenge is that most of the arrow-rs functionality in RecordBatch, for example, assumes unique column names (
find_by_namefor example). I don't think the Arrow spec itself restricts names to be unique, but I think the assumption may run pretty deep in the implementationstv42 commentedon Dec 11, 2023
I assume you mean
column_by_name, https://docs.rs/datafusion/latest/datafusion/common/arrow/record_batch/struct.RecordBatch.html#method.column_by_name?Ecosystem survey:
SQLite picks the first matching column:
Postgres errors at column access time if there are duplicates:
tv42 commentedon Apr 22, 2024
This might be a duplicate of #6543?
alamb commentedon Apr 22, 2024
@tv42 I agree it is certainly related
findepi commentedon Nov 18, 2024
This would be great item in the broader scope of #12723, which intents to make DataFusion Logical Plans be state of the art.