You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat: bucketed scan for native_datafusion Parquet scan #1719
The native_datafusion Parquet scan does not support bucketed scan, and fails most of the tests in Spark's BucketedReadSuite without a fallback. With a bucketed scan, some partitions end up without a file to read so their PartitionedFile is empty.
Describe the potential solution
I don't think DataSourceExec will take no file at construction. We might need to replace that node with a different no-op node that generates an empty data set with the correct schema in the case of a bucketed scan when a partition has no corresponding file.
Interestingly, native_datafusion currently passes the "bucketed table" Comet test. I suspect the Spark SQL tests are doing some bucket pruning, which is where we end up with empty file paths for some of the partitions.
Activity
mbutrovich commentedon May 8, 2025
Interestingly, native_datafusion currently passes the "bucketed table" Comet test. I suspect the Spark SQL tests are doing some bucket pruning, which is where we end up with empty file paths for some of the partitions.