You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
single_distinct_to_groupby panic when group by expr is a binaryExpr #2994
Thanks for your reproducer @jiacai2050. Here are some investigations:
Looks like the single_distinct_to_groupby optimizer doesn't consider the original GROUP BY contains expression like a+1 or b%2. It copies GROUP BYs to the inner subquery, which may cause two problems:
Schema doesn't match. The schema of subquery will become something like #table.a + Int64(1). And thus the outer GROUP BY cannot get a column a from it.
Duplicated evaluation. Expressions in GROUP BY cannot be calculated twice.
To fix this, we need to alias the "original" GROUP BY in the subquery and refer to that aliased column in outer. E.g.:
Before optimize
SELECT id+1, count(distinct(bank_account))
FROM users
GROUP BY id+1;
After optimize
SELECT group_by_alias1 as"id+1", count(bank_account)
FROM (
SELECT id+1as group_by_alias1, bank_account
FROM users
GROUP BY id+1, bank_account
)
GROUP BY group_by_alias1;
It seems a little strange that self.name and self.get_type have different behavior.
Expr::name() is nothing more than a string concat, it won't find whether this column exists in schema. In other words, it won't return an error due to the wrong schema or column:
fn create_name(e:&Expr,input_schema:&DFSchema) -> Result<String>{match e {Expr::Alias(_, name) => Ok(name.clone()),Expr::Column(c) => Ok(c.flat_name()),Expr::Literal(value) => Ok(format!("{:?}", value)),Expr::BinaryExpr{ left, op, right } => {let left = create_name(left, input_schema)?;let right = create_name(right, input_schema)?;Ok(format!("{} {} {}", left, op, right))}
.. => {}}
But get_type and get_nullable can't. The type and nullable info are stored in schema.
BTW, It looks like the input_schema is not used in this function. Maybe we can remove it.
I didn't follow this entire conversation in detail but " adding alias to group by exprs is the way to fix the bug, so you can fix it first." sounds right to me as well
I can't recall the nuances of Expr::name but it may be from before the time we had reasonable relation support (aka there could well be a bug there)
Activity
waynexia commentedon Aug 2, 2022
Thanks for your reproducer @jiacai2050. Here are some investigations:
Looks like the
single_distinct_to_groupbyoptimizer doesn't consider the originalGROUP BYcontains expression likea+1orb%2. It copiesGROUP BYs to the inner subquery, which may cause two problems:#table.a + Int64(1). And thus the outerGROUP BYcannot get a columnafrom it.GROUP BYcannot be calculated twice.To fix this, we need to alias the "original"
GROUP BYin the subquery and refer to that aliased column in outer. E.g.:I'll submit a fix around this or next week.
jiacai2050 commentedon Aug 2, 2022
alias is one potential solution, when I debug this issue, I find something strange:
datafusion panic at
self.get_type()notself.name(), if the rewritten schema is incomplete, why wouldself.namesuccess?It seems a little strange that
self.nameandself.get_typehave different behavior.waynexia commentedon Aug 3, 2022
Expr::name()is nothing more than a string concat, it won't find whether this column exists in schema. In other words, it won't return an error due to the wrong schema or column:But
get_typeandget_nullablecan't. The type and nullable info are stored in schema.BTW, It looks like the
input_schemais not used in this function. Maybe we can remove it.jiacai2050 commentedon Aug 3, 2022
The comment saying that the name is based on schema, but this code is introduced at 2020/10, maybe @andygrove @alamb could share some ideas whether the schema is needed here.
Anyway, I think adding alias to group by exprs is the way to fix the bug, so you can fix it first.
alamb commentedon Aug 3, 2022
I didn't follow this entire conversation in detail but " adding alias to group by exprs is the way to fix the bug, so you can fix it first." sounds right to me as well
I can't recall the nuances of
Expr::namebut it may be from before the time we had reasonable relation support (aka there could well be a bug there)