Skip to content

move projection handling into FileSource - #18627

Merged
adriangb merged 5 commits into
apache:mainfrom
pydantic:projection-source
Nov 27, 2025
Merged

adriangb merged 5 commits into
apache:mainfrom
pydantic:projection-source

Conversation

@adriangb

@adriangb adriangb commented Nov 11, 2025 •

Copy link
Copy Markdown
Contributor

This moves ownership of projections from FileScanConfig into FileSource.
Notably we do not do anything special with this in Parquet just yet: I leave it for a followup to actually use the projection expressions instead of column indices to e.g. generate the Parquet ProjectionMask directly from expressions (in particular to select leaves instead of roots for struct and variant access).

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api change Changes the API exposed to users of the crate catalog Related to the catalog crate core Core DataFusion crate datasource Changes to the datasource crate documentation Improvements or additions to documentation physical-expr Changes to the physical-expr crates physical-plan Changes to the physical-plan crate proto Related to proto crate sqllogictest SQL Logic Tests (.slt) substrait Changes to the substrait crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants