Skip to content

Pass PartitionedFile into FileSource for late file stats based pruning #16000

Closed

Description

@adriangb

Is your feature request related to a problem or challenge?

As we continue to make progress landing dynamic filters it opens up the opportunity for new optimizations.

This one deals with late evaluation of file-level statistics.

In particular, we may have file level statics available at planning time (see datafusion.execution.collect_statistics - our system does a similar thing in a different way).
Before dynamic filters there was no point in re-evaluating these right before scanning a file but now it's possible that e.g. a TopK operator passed down a ts > '2025-05-08T00:00:00Z' filter -> we may be able to exclude the entire file based on this filter + file level statistics -> we avoid reading any Parquet metadata, etc.

In particular, change:

trait FileSource {
  fn open(file_meta: FileMeta) ...

To:

trait FileSource {
  fn open(file_meta: FileMeta, file: &PartitionedFile) ...

And call it as so from here:

https://github.com/pydantic/datafusion/blob/649851d59cdac80fcae51d66f82f1b47d2aaa3b4/datafusion/datasource/src/file_stream.rs#L129

Then we can implement PruningStatitics for Statistics et. voilá!

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

Activity

  1. adriangb commented on May 9, 2025

    @adriangb
    ContributorAuthor
  2. alamb commented on May 9, 2025

    @alamb
    Contributor

    I think the idea of passing the partitioned file (with the optional per-file statistics) is an excellent idea 👍

  3. adriangb commented on May 10, 2025

    @adriangb
    ContributorAuthor

    POC at #16014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions