You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Pass PartitionedFile into FileSource for late file stats based pruning #16000
Is your feature request related to a problem or challenge?
As we continue to make progress landing dynamic filters it opens up the opportunity for new optimizations.
This one deals with late evaluation of file-level statistics.
In particular, we may have file level statics available at planning time (see datafusion.execution.collect_statistics - our system does a similar thing in a different way).
Before dynamic filters there was no point in re-evaluating these right before scanning a file but now it's possible that e.g. a TopK operator passed down a ts > '2025-05-08T00:00:00Z' filter -> we may be able to exclude the entire file based on this filter + file level statistics -> we avoid reading any Parquet metadata, etc.
Activity
adriangb commentedon May 9, 2025
cc @berkaysynnada @alamb
alamb commentedon May 9, 2025
I think the idea of passing the partitioned file (with the optional per-file statistics) is an excellent idea 👍
adriangb commentedon May 10, 2025
POC at #16014