You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Fix for issue #6595 has broken existing working queries ("Schema error: No field named ...") #6897
After upgrading to Datafusion 27.0.0 we noticed some of our regression tests were failing. We bisected the commit that introduced the break to 36123ee, which is the fix for #6595.
To Reproduce
The attached zip file contains a parquet file. To reproduce the issue in datafusion-cli, run:
> create external table t0 stored as parquet location 'test_data.parquet';
> SELECT "day" AS "date", count(distinct "direction") AS "num_directions" FROM t0 GROUP BY "day" ORDER BY t0."day" ASC;
In datafusion 26 and earlier, this will generate a result. In datafusion 27, it generates this error message:
DataFusion CLI v26.0.0
❯ SELECT "day" AS "date", count(distinct "direction") AS "num_directions" FROM 'test_data.parquet' GROUP BY "day" ORDER BY "day" ASC;
+---------------------+----------------+
| date | num_directions |
+---------------------+----------------+
| 2011-09-09T00:00:00 | 2 || 2011-09-10T00:00:00 | 2 |
...
| 2018-04-14T00:00:00 | 2 || 2018-04-15T00:00:00 | 2 |
+---------------------+----------------+
81 rows in set. Query took 0.024 seconds.
❯
27.0.0 fails
DataFusion CLI v27.0.0
❯ SELECT "day" AS "date", count(distinct "direction") AS "num_directions" FROM 'test_data.parquet' GROUP BY "day" ORDER BY "day" ASC;
Optimizer rule 'simplify_expressions' failed
caused by
Schema error: No field named "test_data.parquet".day. Valid fields are "test_data.parquet.day", "COUNT(DISTINCT test_data.parquet.direction)".
❯
main passes:
$ git checkout main
Already on 'main'
Your branch is up to date with 'apache/main'.
$ CARGO_TARGET_DIR=/Users/alamb/Software/target-df cargo run
Finished dev [unoptimized + debuginfo] target(s) in 0.27s
Running `/Users/alamb/Software/target-df/debug/datafusion-cli`
DataFusion CLI v27.0.0
❯ SELECT "day" AS "date", count(distinct "direction") AS "num_directions" FROM 'test_data.parquet' GROUP BY "day" ORDER BY "day" ASC;
+---------------------+----------------+
| date | num_directions |
+---------------------+----------------+
| 2011-09-09T00:00:00 | 2 || 2011-09-10T00:00:00 | 2 |
...
| 2018-04-14T00:00:00 | 2 || 2018-04-15T00:00:00 | 2 |
+---------------------+----------------+
81 rows in set. Query took 0.027 seconds.
changed the title [-]Fix for issue #6595 has broken existing working queries[/-][+]Fix for issue #6595 has broken existing working queries ("Schema error: No field named ...")[/+]on Jul 10, 2023
Activity
alamb commentedon Jul 10, 2023
I believe this is similar to #6790 which was since fixed in #6827. I will verify that this is the same thing (we saw a similar error in IOx)
alamb commentedon Jul 10, 2023
I have verified this has been fixed on master (aka what will be released in DataFusion
28.0.0).BTW I added new test coverage in #6836 so that we don't break this again by accident.
Since it is a regression I would be willing to create a patch release (
27.0.1) with the fix if that would be helpful for othersUsing this query (thanks for the reproducer @maxburke 🙏 )
26.0.0works27.0.0failsmainpasses:alamb commentedon Jul 10, 2023
cc @jackwener
[-]Fix for issue #6595 has broken existing working queries[/-][+]Fix for issue #6595 has broken existing working queries ("Schema error: No field named ...")[/+]