You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Review use of logical expressions in physical AggregateFunctionExpr #11359
Is your feature request related to a problem or challenge?
DataFusion 40.0.0 added a new logical_args: Vec<Expr> field to AggregateFunctionExpr, which seems confusing, and there is no documentation in this struct that explains what this field is used for.
In DataFusion Comet, we do not use DataFusion's logical plan or expressions because we are translating an Apache Spark physical plan into a DataFusion physical plan and therefore we have no logical expressions to pass into this new field.
I think at a minimum we should add some documentation around this new feature.
Reacted by Jay Zhan, Dharan Aditya, Edward Jones and Huaijin
@ozankabak had asked for whether there is anyway to entirely getting rid of logical expressions in discord, so I think we can review about the challenge I had before.
The reason why there are logical expressions in creeate_aggregate_expr is for customizing Accumulator. We can have different kind of accumulator based on the function's arguments. Ideally we should check with PhysicalExpr, but unfortunately, given the current design, we are not able to introduce physical expressions in AggregateUDFImpl trait, because we want to avoid adding dependency of physical-expr crate to datafusion-expr.
I think that it is also the main reason that blocking us. I propose to redesign about the role of each crate. To able to deal with physical concept for AggregateUDFImpl, we break physical-expr into two level. One is physical-expr-common, and another is physical-expr. The same applies to logical expr, datafusion-expr-common and datafusion-expr. common crate is the higher level crate so that we can add the dependency of physical-expr-common into datafusion-expr
expr-common: Things other than Expr and LogicalPlan can place it here expr: Mainly for Expr and LogicalPlan. Import functions-aggregate-common for UDAF. physical-common: Physical expr trait or other common things. Similar to what it is now except physical-expr like Column, Cast, Literal which should be moved to physical-expr physical-expr: Physical Expr are here + Column, Cast, Literal functions-aggregate-common: Import physical-common and expr-common, for other users to build their own udaf. functions-aggregate: datafusion builtin functions
With this approach, function like limited_convert_logical_expr_to_physical_expr is no longer needed after this change.
We have at least two such functions and it would be great to arrive at a design that eliminates such functions.
Conceptually, we should have more info at the physical level (relative to logical level), so a refactor that makes AggregateFunctionExpr and likewise structs independent of logicals should be possible. I get the feeling that our dependency structure is currently more complex than it needs to be.
I agree with @jayzhan211 that the core of the problem is that the user defined API for aggregates is in datafusion_expr so can only use Expr but is invoked / instantiated as part of the physical plan.
However, the same basic problem could be claimed for ScalarUDFImpl and WindowUDFImpl 🤔
It does feel like the way out of this is to make the spit between API and implementation more explicit.
Maybe instead of expr-common maybe we could call it expr-api and physical-common --> physical-api
So like
expr-api: Traits like UDFetc other than Expr and LogicalPlan can place it here
expr: (existing crate) Mainly for Expr and LogicalPlan. Import functions-aggregate-common for UDAF.
physical-api: Physical expr trait or other common things. Similar to what it is now except physical-expr like Column, Cast, Literal which should be moved to physical-expr
physical-expr-common Physical Expr are here + Column, Cast, Literal
`aggregate-api: Import physical-api and expr-api -- TBD what does this have? ScalarUDAF? Accumulator?
I agree we should document what the fields are used for now
I personally recommend we finish #8708 before we try to do some other crate refactor. We are close with that one and once we have all the aggregates going through the same APIs I think we'll be in a better position to split things apart
Reacted by Jay Zhan, Mehmet Ozan Kabak and Edward Jones
I think it is possible to have expr-api crate that contains ContextProvider, TableSource, FunctionRegistry, AggregateUDF, WindowUDF, and maybe ScalarUDF (this is not necessary but it would be nice to be close to other two functions)
physical-api: Physical expr trait or other common things. Similar to what it is now except physical-expr like Column, Cast, Literal which should be moved to physical-expr
physical-expr-common Physical Expr are here + Column, Cast, Literal
Not sure about the reason for physical-expr-common crate that contains Physical Expr, that is almost 60 ~ 80 % of the things in physical-expr, I perfer moving them back 🤔
physical-api / physical-expr-common for PhysicalExpr and PhysicalSortExpr and utils function for physical expr.
The original idea of functions-aggregate-common is crate for AggregateUDFImpl, we can
Keep ContextProvider, FunctionRegistry, TableSource and UDF,UDAF,UDWF in the same crate. The dependency is expr-api -> expr
UDF,UDAF,UDWF in expr-functions and move ContextProvider, FunctionRegistry and TableSource to a crate like expr-provider / expr-registry. The dependency is expr-provider -> expr-functions -> expr
I plan to pull out aggregate function to functions-aggregate-common (the role of expr-functions in the above graph but narrow to aggregate only) as the first step. Ok, this doesn't work 😆 , AggregateFunction and Expr should belong to the same crate.
Expr::AggregateFunction(AggregateFunction)pubstructAggregateFunction{/// Name of the functionpubfunc:Arc<crate::AggregateUDF>,
..
}pubstructAggregateUDF{inner:Arc<dynAggregateUDFImpl>,}
I found AggregateUDFImpl is now tightly coupled with Expr unlike the status I got in #10327, thus not possible to eliminate the dependency of Expr. Mainly due to fn call() -> Expr
The crate graph is now like
Since we would like to import PhysicalExpr for AggregateUDFImpl, and sadly it seems to be tightly coupled with Expr.
Therefore, we come out several common-level crate, those crate has no dependencies on Expr. physical-expr-common are common things about physical-expr concept. Mainly PhysicalExpr, PhysicalSortExpr.
Their dependencies are pull out from expr to expr-common. functions-aggregate-common are more aggregate-specific
Physical expr like Column, Cast, Literal are moved back to physical-expr for now. AggregateFunctionExpr, AggregateExprBuilder are moved to physical-expr-functions-aggregate, they have depdency on Expr. It's level is similar to physical-expr, but more aggregate specific.
Activity
jayzhan211 commentedon Jul 10, 2024
@ozankabak had asked for whether there is anyway to entirely getting rid of logical expressions in discord, so I think we can review about the challenge I had before.
The reason why there are logical expressions in
creeate_aggregate_expris for customizingAccumulator. We can have different kind of accumulator based on the function's arguments. Ideally we should check with PhysicalExpr, but unfortunately, given the current design, we are not able to introduce physical expressions inAggregateUDFImpltrait, because we want to avoid adding dependency ofphysical-exprcrate todatafusion-expr.I think that it is also the main reason that blocking us. I propose to redesign about the role of each crate. To able to deal with physical concept for
AggregateUDFImpl, we breakphysical-exprinto two level. One isphysical-expr-common, and another isphysical-expr. The same applies to logical expr,datafusion-expr-commonanddatafusion-expr.common crateis the higher level crate so that we can add the dependency ofphysical-expr-commonintodatafusion-exprThe crate graph is like
graph TD; functions-aggregates-common-->expr-common; functions-aggregates-common-->physical-expr-common; functions-aggregates-->functions-aggregates-common; functions-aggregates-->expr; physical-expr-common-->expr-common; expr-->expr-common; expr-->functions-aggregates-common; physical-expr-->physical-expr-common; core-->functions-aggregates; core-->physical-expr; third-parties-aggregate --> functions-aggregates-common;expr-common: Things other thanExprandLogicalPlancan place it hereexpr: Mainly forExprandLogicalPlan. Importfunctions-aggregate-commonfor UDAF.physical-common: Physical expr trait or other common things. Similar to what it is now except physical-expr like Column, Cast, Literal which should be moved to physical-exprphysical-expr: Physical Expr are here + Column, Cast, Literalfunctions-aggregate-common: Importphysical-commonandexpr-common, for other users to build their own udaf.functions-aggregate: datafusion builtin functionsThe more detail of discussion before is in #10074
With this approach, function like
limited_convert_logical_expr_to_physical_expris no longer needed after this change.@alamb I think we can review about this idea again, the previous concern is that
I think it is not an issue anymore.
ozankabak commentedon Jul 10, 2024
We have at least two such functions and it would be great to arrive at a design that eliminates such functions.
Conceptually, we should have more info at the physical level (relative to logical level), so a refactor that makes
AggregateFunctionExprand likewise structs independent of logicals should be possible. I get the feeling that our dependency structure is currently more complex than it needs to be.jayzhan211 commentedon Jul 11, 2024
I think we tend to split crate aggressively so others can only import the necessary crate they need.
ozankabak commentedon Jul 11, 2024
Splitting crates when beneficial is great, maybe we haven't arrived at the best design in terms of structure yet (as evidenced by this issue).
I think we can use your previous work as a starting point to improve the structure and also resolve this current issue 🚀
alamb commentedon Jul 12, 2024
I agree with @jayzhan211 that the core of the problem is that the user defined API for aggregates is in datafusion_expr so can only use
Exprbut is invoked / instantiated as part of the physical plan.However, the same basic problem could be claimed for
ScalarUDFImplandWindowUDFImpl🤔It does feel like the way out of this is to make the spit between API and implementation more explicit.
Maybe instead of
expr-commonmaybe we could call itexpr-apiandphysical-common-->physical-apiSo like
expr-api: Traits likeUDFetc other than Expr and LogicalPlan can place it hereexpr: (existing crate) Mainly for Expr and LogicalPlan. Import functions-aggregate-common for UDAF.physical-api: Physical expr trait or other common things. Similar to what it is now except physical-expr like Column, Cast, Literal which should be moved to physical-exprphysical-expr-commonPhysical Expr are here + Column, Cast, Literalfunctions-aggregate: datafusion builtin functionsalamb commentedon Jul 12, 2024
I agree we should document what the fields are used for now
I personally recommend we finish #8708 before we try to do some other crate refactor. We are close with that one and once we have all the aggregates going through the same APIs I think we'll be in a better position to split things apart
jayzhan211 commentedon Jul 13, 2024
Cool, maybe I could think about pulling down functions trait from
exprinstead of pulling up common things toexpr-common🤔jayzhan211 commentedon Jul 13, 2024
I think it is possible to have
expr-apicrate that containsContextProvider,TableSource,FunctionRegistry,AggregateUDF,WindowUDF, and maybeScalarUDF(this is not necessary but it would be nice to be close to other two functions)Not sure about the reason for
physical-expr-commoncrate that contains Physical Expr, that is almost 60 ~ 80 % of the things inphysical-expr, I perfer moving them back 🤔physical-api / physical-expr-commonforPhysicalExprandPhysicalSortExprand utils function for physical expr.The original idea of
functions-aggregate-commonis crate forAggregateUDFImpl, we canContextProvider,FunctionRegistry,TableSourceand UDF,UDAF,UDWF in the same crate. The dependency isexpr-api -> exprexpr-functionsand moveContextProvider,FunctionRegistryandTableSourceto a crate likeexpr-provider / expr-registry. The dependency isexpr-provider -> expr-functions -> exprThe crate graph is like
graph TD; expr-api --> physical-expr-common; physical-expr-common --> expr; functions-aggregate --> expr-api; third-parties-aggregate --> expr-api;or
graph TD; expr-provider --> expr-functions; expr-functions --> physical-expr-common; physical-expr-common --> expr; functions-aggregate --> expr-functions; third-parties-aggregate --> expr-functions;jayzhan211 commentedon Aug 5, 2024
I plan to pull out aggregate function to. Ok, this doesn't work 😆 , AggregateFunction and Expr should belong to the same crate.functions-aggregate-common(the role of expr-functions in the above graph but narrow to aggregate only) as the first stepI guess #10327 is the only possible solution
jayzhan211 commentedon Aug 6, 2024
I found
AggregateUDFImplis now tightly coupled withExprunlike the status I got in #10327, thus not possible to eliminate the dependency ofExpr. Mainly due tofn call() -> ExprThe crate graph is now like
Since we would like to import
PhysicalExprforAggregateUDFImpl, and sadly it seems to be tightly coupled withExpr.Therefore, we come out several
common-levelcrate, those crate has no dependencies onExpr.physical-expr-commonare common things aboutphysical-exprconcept. MainlyPhysicalExpr,PhysicalSortExpr.Their dependencies are pull out from
exprtoexpr-common.functions-aggregate-commonare more aggregate-specificPhysical expr like Column, Cast, Literal are moved back to physical-expr for now.
AggregateFunctionExpr,AggregateExprBuilderare moved tophysical-expr-functions-aggregate, they have depdency onExpr. It's level is similar tophysical-expr, but more aggregate specific.graph TD; physical-expr-common --> expr-common; functions-aggregate-common --> physical-expr-common; expr --> functions-aggregate-common; physical-expr-functions-aggregate --> expr; physical-expr --> expr; functions-aggregate --> physical-expr;