Skip to content

feat: give the Iceberg reader dynamic filters to skip row groups - #22

Draft
Perseus wants to merge 2 commits into
anirudh.singh/iceberg-dyn-filtering-3from
anirudh.singh/iceberg-dyn-filtering-4
Draft

Perseus wants to merge 2 commits into
anirudh.singh/iceberg-dyn-filtering-3from
anirudh.singh/iceberg-dyn-filtering-4

Conversation

@Perseus

@Perseus Perseus commented Nov 10, 2026

Copy link
Copy Markdown
Owner

As the reader starts each file, the scan now ANDs the newest dynamic filters onto the file's own filter, so the reader can skip row groups the filters rule out. This covers joins on columns that aren't partition columns, and top-N.

The reader gets some columns wrong, so filters on them are left out: identity partition columns (the partition skipper checks those), columns with a default value, and nested columns. Above 200 listed values, IN lists are dropped and their bounds kept. The reader only gets the filters when row group filtering is on and page skipping is off, because page skipping gets widened columns wrong. Any error leaves the file as it is.

@Perseus
Perseus force-pushed the anirudh.singh/iceberg-dyn-filtering-4 branch from 0bfae70 to c7461f9 Compare November 11, 2026 14:35
@Perseus
Perseus force-pushed the anirudh.singh/iceberg-dyn-filtering-3 branch from 77858c6 to c59f43e Compare November 11, 2026 14:35
@Perseus
Perseus force-pushed the anirudh.singh/iceberg-dyn-filtering-4 branch 11 times, most recently from 0162fe2 to d5bec23 Compare November 17, 2026 18:52
The next commit gives the Iceberg reader the scan's dynamic filters, so it
can skip row groups. The reader only uses the filter stored on each
FileScanTask, and iceberg-rust has no way to change that filter.

and_predicate works around this. It turns the task's filter back into an
unbound predicate, ANDs the new predicate onto it, binds the result and
builds a copy of the task with it. Nothing calls it yet.

The copy sets each of the task's fields by hand. A snapshot test lists the
task's fields, so it fails if iceberg-rust adds a field the copy would
leave out.
With iceberg.dynamic_filtering_enabled on, the scan now gives the reader
the newest dynamic filters as it starts each file. The reader uses them to
skip row groups, and to drop the rows that don't match. This helps joins on
columns that aren't partition columns, and top-N.

The reader checks every row once per IN list value. When the IN lists hold
more than 200 values in total, we leave them out and keep the min/max
bounds.

The reader reads a column the file doesn't store as NULL. So a filter on an
identity partition column, or on a column added later with a default value,
can drop every row of such a file. We accept this for now.

Any error leaves the file's task as it was.
@Perseus
Perseus force-pushed the anirudh.singh/iceberg-dyn-filtering-3 branch from 1b1af3d to 249fd05 Compare November 18, 2026 09:44
@Perseus
Perseus force-pushed the anirudh.singh/iceberg-dyn-filtering-4 branch from d5bec23 to f0b5fb3 Compare November 18, 2026 09:44

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant