You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
A problem about the projection_push_down optimizer gathers valid columns #1312
Describe the bug
why gather all columns needed for expressions in this Aggregate by using aggr_expr, rather than schema.
An error occurs when the schema is generated by a custom optimizer and does not match the column name in aggr_expr.
I think this is probably written with the assumption that schema is always in sync with aggr_expr. @ic4y could you give a more concrete example on how a custom optimizer would change the schema without updating aggr_expr?
@houqp
Like the following, I added group by for distinct Aggregate, and then deleted distinct in count. At this time, aggr_expr has changed, but I don’t want to rewrite the schema, because subsequent Plan Nodes will use this schema like ProjectionNode
Projection: #COUNT(DISTINCT lineorder_flat.lo_orderkey) AS a [a:UInt64;N]
Aggregate: groupBy=[[]], aggr=[[COUNT(DISTINCT #lineorder_flat.lo_orderkey)]] [COUNT(DISTINCT lineorder_flat.lo_orderkey):UInt64;N]
TableScan: lineorder_flat projection=Some([0]) [lo_orderkey:Int64;N]
-------------------custom optimizer------------------------------------------
Projection: #COUNT(DISTINCT lineorder_flat.lo_orderkey) AS a [a:UInt64;N]
Aggregate: groupBy=[[]], aggr=[[COUNT(#lineorder_flat.lo_orderkey)]] [COUNT(DISTINCT lineorder_flat.lo_orderkey):UInt64;N]
Aggregate: groupBy=[[#lineorder_flat.lo_orderkey]], aggr=[[]] [lo_orderkey:Int64;N]
TableScan: lineorder_flat projection=Some([0]) [lo_orderkey:Int64;N]
Activity
houqp commentedon Nov 16, 2021
I think this is probably written with the assumption that schema is always in sync with aggr_expr. @ic4y could you give a more concrete example on how a custom optimizer would change the schema without updating aggr_expr?
ic4y commentedon Nov 16, 2021
@houqp
Like the following, I added group by for distinct Aggregate, and then deleted distinct in count. At this time, aggr_expr has changed, but I don’t want to rewrite the schema, because subsequent Plan Nodes will use this schema like ProjectionNode
alamb commentedon Nov 17, 2021
This sounds very similar to #1316, which @viirya is working on in #1319 I believe
ic4y commentedon Nov 22, 2021
I used projection output relation alias to solve this problem
Support arrays_overlap function (apache#1312)