You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Currently a named expression / alias can be encoded by wrapping an existing Expr in an Expr::Alias.
However, this has some side effects:
We can encode expressions that don't make sense, such as (1 + 1 AS x) AS y or (1 AS X) + (1 AS Y) (in SQL it's disallowed by the parser, but it's just to show it is possible to do this in the DataFrame API or the raw API. In dataframes we could do things like expr.alias("x").alias("y").
Code dealing with Expr always have to deal with the Alias case, even when it doesn't care. This could lead to more complex code or even subtle bugs.
Describe the solution you'd like
Create a new struct, NamedExpr that is used to refer to a named expression
structNamedExpr{/// Alias or generated namename:String,/// The expressionexpr:Expr,}
The NamedExpr now can be used inside projections, aggregates, etc.
The function alias on DataFrame should return an NamedExpr
Add an impl From<Expr> for NamedExpr that generates a name.
Some functions can accept Into to keep being ergonomic to use.
Describe alternatives you've considered
Additional context
Reacted by Piotr Findeisen
Reacted by Andrew Lamb, QP Hou, Jason Tianyi Wang, Jay Zhan and Piotr Findeisen
It's not entirely clear to me how we are going to propagate it through out the code base. But if we can pull it off, that would make the code more readable and maintainable for sure 👍
The idea is that any expression within DataFusion receives a name, so the nodes in a LogicalPlan use the NamedExpr type. The name in this type is either the alias for the expression if any or the generated name, so it doesn't need to be optional.
In addition, I find some Filter Expr use Alias after rewrite which make logicplan can be connected after rewrited.
It's related with #1319#1316
It mayn't a good way by using alias. Because we must add alias after write Expr.
BTW, projection is same. we must add alias after rewrite plan
I think it is a future ticket.
I spent some time on this and I am no longer sure that this change makes sense. We have a lot of optimizer rules that recurse through expression trees rewriting them and if we move the Alias variant out of the Expr enum it complicates things because we still need to recurse into the expression contained within the alias. It doesn't feel like the effort here justifies the benefit, IMO.
There are other Expr variants that only make sense in certain contexts as well. We can't have an Expr::Sort in a projection, for example. Maybe we just need some validation rules to ensure plans make sense and better utility methods for dealing with alias in cases where it is not valid?
I run into this recently. When constructing VALUES via API, the Alias expression can be thought to allow aliasing the VALUES columns, but this didn't work, alias got (silently) ignored. From such "unpleasant surprise" perspective, I would consider this as a bug.
To me, alias is not an expression at all. It's a feature of a select clause and would be best modeled as such.
There are other Expr variants that only make sense in certain contexts as well. We can't have an Expr::Sort in a projection, for example.
That's a good point and good example.
To me, in ORDER BY <expr> [ASC/DESC] [NULLS FIRST/LAST], the <expr> part is an expression (any expression), and the other attributes (asc/desc, nulls first/last) are attributes of the sorting. They don't have to be modeled as "an expression".
I run into this recently. When constructing VALUES via API, the Alias expression can be thought to allow aliasing the VALUES columns, but this didn't work, alias got (silently) ignored. From such "unpleasant surprise" perspective, I would consider this as a bug.
I agree it sounds like a bug to me
To me, alias is not an expression at all. It's a feature of a select clause and would be best modeled as such.
Theoretically I think this makes sense
One thing that is different about DataFusion than other systems is that the schemas are based on name (rather than column order) and the names of the columns are derived from the expressions
So the point is that an expression like a + 5 + 1 will result in a column named approximately "a + 5 + 1" -- so when the optimizer passes rewrite expressions they need to preserve the output name, and they do so using Expr::Alias
So in this case
a +5+1
Becomes
a +6as"a + 5 + 1"
That rewrite can happen in multiple parts of the plan, not just the select list
There are likely other ways to handle this than Expr::Alias (a separate list on Projection, for example) but I do think alias is used widely
There are other Expr variants that only make sense in certain contexts as well. We can't have an Expr::Sort in a projection, for example.
That's a good point and good example. To me, in ORDER BY <expr> [ASC/DESC] [NULLS FIRST/LAST], the <expr> part is an expression (any expression), and the other attributes (asc/desc, nulls first/last) are attributes of the sorting. They don't have to be modeled as "an expression".
I think Sort would be an easier thing to remove / fix -- Expr::Sort as an expression is also bad as it means the signatures of fn order_by(...) are in terms of Expr, meaning the compiler can't ensure you are actually passing Expr::Sort when needed
I think Sort would be an easier thing to remove / fix -- Expr::Sort as an expression is also bad as it means the signatures of fn order_by(...) are in terms of Expr, meaning the compiler can't ensure you are actually passing Expr::Sort when needed
Indeed. There are a few places where sort expression needs to be special cased.
There are likely other ways to handle this than Expr::Alias (a separate list on Projection, for example) but I do think alias is used widely
I see the point. I would be tempted to go into direction where names are explicit and expressions are just expressions.
Ie to model projects as map<String /* name to assign */, Expr /* value to calculate */>
one thing is how easy it is to get there (i get that likely pretty hard per "alias is used widely")
and the other is whether we would want to get there at all (is this the right direction)
one thing is how easy it is to get there (i get that likely pretty hard per "alias is used widely")
I agree
and the other is whether we would want to get there at all (is this the right direction)
I don't have any strong feeling in this regard -- in general Expr::Alias seems to work ok, though there are some places where having to handle it (in filters, etc) have caused some issues in the past (which is why there is Expr::unalias and various other methods)
Besides fixing some problems (described in this issue + #1468 (comment)), separating "expression" and "named expression" could result in lower mental overhead involved when dealing with the code, given that in many contexts expressions don't have & don't need names (eg VALUES, ORDER BY, function inputs).
I think Sort would be an easier thing to remove / fix -- Expr::Sort as an expression is also bad as it means the signatures of fn order_by(...) are in terms of Expr, meaning the compiler can't ensure you are actually passing Expr::Sort when needed
filed #12193 for this
and created a PR removing Expr::Sort (#12177)
I'm confused about the relation in Alias. What is it for? Could we remove it? While digging into #12184, I think we don't need to keep relation in Alias, instead get the relation from expr (Specifically Expr::Column) if needed.
I'm confused about the relation in Alias. What is it for? Could we remove it? While digging into #12184, I think we don't need to keep relation in Alias, instead get the relation from expr (Specifically Expr::Column) if needed.
It may be historical -- Expr::Alias is quite old code I think
Schema / DFSchema has dual purpose and this is related to Expr::Alias existence and handling.
One necessary purpose is understanding source tables, and identifier resolution in the SQL queries - a necessary job to be done once SQL syntax tree (AST) is produced by the parser.
The other purpose is for column/symbol/variable resolution between the logical plan components. Some engines (like Trino) use separate abstraction for that purpose. Reusing schema here makes it very hard to solve #13476, #6543 (see WIP #13489 (comment))
I know addressing this is hard, but in 5 year from now perspective, would we prefer these issues to be still with us? Or would we prefer to address them sooner or later?
I think we should refactor LogicalPlans not to use aliases and column references at all. This can fall under #12723. We should find a way to do this incrementally, and I believe there is a way to do so.
I know addressing this is hard, but in 5 year from now perspective, would we prefer these issues to be still with us? Or would we prefer to address them sooner or later?
Well I am always for improving the situation!
I think we should refactor LogicalPlans not to use aliases and column references at all. This can fall under #12723. We should find a way to do this incrementally, and I believe there is a way to do so.
I agree -- doing it incrementally is the way to go and the only challenge will be to gather enough people / time to help make it happen.
Activity
Dandandan commentedon Dec 20, 2021
FYI @alamb @houqp
alamb commentedon Dec 21, 2021
I think
NamedExprsounds like a great idea, personally. 👍xudong963 commentedon Dec 21, 2021
Make sense to me @Dandandan
houqp commentedon Dec 24, 2021
It's not entirely clear to me how we are going to propagate it through out the code base. But if we can pull it off, that would make the code more readable and maintainable for sure 👍
jackwener commentedon Apr 7, 2022
I think it is valuable, there are some case code just for handling the Alias::Expr, I'm willing to do this task.
I will push a draft PR for this issue this weekend.
jackwener commentedon Apr 9, 2022
@Dandandan . Hi, I'm a little confused about the details.
Specifically how to unify
NamedExprandExpr. I think option is a good way.Now alias in
Expr:Expr is enum as
After use
NamedExpr,The structure is like thisCan we use this struct to present it?
Dandandan commentedon Apr 9, 2022
Hi @jackwener .
The idea is that any expression within DataFusion receives a name, so the nodes in a LogicalPlan use the
NamedExprtype. The name in this type is either the alias for the expression if any or the generated name, so it doesn't need to be optional.jackwener commentedon Apr 10, 2022
😅I found that it isn't easy problem, I need to think more about it.
Expris used in many places.jackwener commentedon Apr 10, 2022
Aliasesin SQL is just used inTableandColumnin thefromClauseandselectClause. So, likeAggregate,aggr_exprisNamedExpr, butgroup_exprisn't.jackwener commentedon Apr 10, 2022
In addition, I find some
Filter ExpruseAliasafter rewrite which makelogicplancan be connected after rewrited.It's related with #1319 #1316
It mayn't a good way by using alias. Because we must add alias after write
Expr.BTW,
projectionis same. we must add alias after rewriteplanI think it is a future ticket.
andygrove commentedon May 3, 2022
I plan on working on this since I think it will solve some problems I am running into when working with expressions and schemas.
andygrove commentedon May 5, 2022
I spent some time on this and I am no longer sure that this change makes sense. We have a lot of optimizer rules that recurse through expression trees rewriting them and if we move the
Aliasvariant out of theExprenum it complicates things because we still need to recurse into the expression contained within the alias. It doesn't feel like the effort here justifies the benefit, IMO.There are other
Exprvariants that only make sense in certain contexts as well. We can't have anExpr::Sortin a projection, for example. Maybe we just need some validation rules to ensure plans make sense and better utility methods for dealing with alias in cases where it is not valid?findepi commentedon Aug 24, 2024
I run into this recently. When constructing VALUES via API, the Alias expression can be thought to allow aliasing the VALUES columns, but this didn't work, alias got (silently) ignored. From such "unpleasant surprise" perspective, I would consider this as a bug.
To me, alias is not an expression at all. It's a feature of a select clause and would be best modeled as such.
That's a good point and good example.
To me, in
ORDER BY <expr> [ASC/DESC] [NULLS FIRST/LAST], the<expr>part is an expression (any expression), and the other attributes (asc/desc, nulls first/last) are attributes of the sorting. They don't have to be modeled as "an expression".alamb commentedon Aug 24, 2024
I agree it sounds like a bug to me
Theoretically I think this makes sense
One thing that is different about DataFusion than other systems is that the schemas are based on name (rather than column order) and the names of the columns are derived from the expressions
So the point is that an expression like
a + 5 + 1will result in a column named approximately"a + 5 + 1"-- so when the optimizer passes rewrite expressions they need to preserve the output name, and they do so usingExpr::AliasSo in this case
Becomes
That rewrite can happen in multiple parts of the plan, not just the select list
There are likely other ways to handle this than
Expr::Alias(a separate list onProjection, for example) but I do think alias is used widelyI think Sort would be an easier thing to remove / fix --
Expr::Sortas an expression is also bad as it means the signatures offn order_by(...)are in terms ofExpr, meaning the compiler can't ensure you are actually passingExpr::Sortwhen neededfindepi commentedon Aug 25, 2024
Indeed. There are a few places where sort expression needs to be special cased.
I see the point. I would be tempted to go into direction where names are explicit and expressions are just expressions.
Ie to model projects as
map<String /* name to assign */, Expr /* value to calculate */>one thing is how easy it is to get there (i get that likely pretty hard per "alias is used widely")
and the other is whether we would want to get there at all (is this the right direction)
alamb commentedon Aug 25, 2024
I agree
I don't have any strong feeling in this regard -- in general
Expr::Aliasseems to work ok, though there are some places where having to handle it (in filters, etc) have caused some issues in the past (which is why there isExpr::unaliasand various other methods)findepi commentedon Aug 25, 2024
Besides fixing some problems (described in this issue + #1468 (comment)), separating "expression" and "named expression" could result in lower mental overhead involved when dealing with the code, given that in many contexts expressions don't have & don't need names (eg VALUES, ORDER BY, function inputs).
findepi commentedon Aug 27, 2024
filed #12193 for this
and created a PR removing
Expr::Sort(#12177)jayzhan211 commentedon Aug 27, 2024
I'm confused about the
relationinAlias. What is it for? Could we remove it? While digging into #12184, I think we don't need to keeprelationinAlias, instead get the relation fromexpr(Specifically Expr::Column) if needed.alamb commentedon Sep 5, 2024
It may be historical --
Expr::Aliasis quite old code I thinkfindepi commentedon Nov 20, 2024
Schema / DFSchema has dual purpose and this is related to Expr::Alias existence and handling.
One necessary purpose is understanding source tables, and identifier resolution in the SQL queries - a necessary job to be done once SQL syntax tree (AST) is produced by the parser.
The other purpose is for column/symbol/variable resolution between the logical plan components. Some engines (like Trino) use separate abstraction for that purpose. Reusing schema here makes it very hard to solve #13476, #6543 (see WIP #13489 (comment))
I know addressing this is hard, but in 5 year from now perspective, would we prefer these issues to be still with us? Or would we prefer to address them sooner or later?
I think we should refactor LogicalPlans not to use aliases and column references at all. This can fall under #12723. We should find a way to do this incrementally, and I believe there is a way to do so.
alamb commentedon Nov 21, 2024
Well I am always for improving the situation!
I agree -- doing it incrementally is the way to go and the only challenge will be to gather enough people / time to help make it happen.
kartik00052 commentedon Aug 11, 2026
is this issue still open.....????