Repository navigation
Conversation
Perseus
force-pushed
the
anirudh.singh/iceberg-dyn-filtering-4
branch
from
November 11, 2026 14:35
0bfae70 to
c7461f9
Compare
Perseus
force-pushed
the
anirudh.singh/iceberg-dyn-filtering-3
branch
from
November 11, 2026 14:35
77858c6 to
c59f43e
Compare
Perseus
force-pushed
the
anirudh.singh/iceberg-dyn-filtering-4
branch
11 times, most recently
from
November 17, 2026 18:52
0162fe2 to
d5bec23
Compare
The next commit gives the Iceberg reader the scan's dynamic filters, so it can skip row groups. The reader only uses the filter stored on each FileScanTask, and iceberg-rust has no way to change that filter. and_predicate works around this. It turns the task's filter back into an unbound predicate, ANDs the new predicate onto it, binds the result and builds a copy of the task with it. Nothing calls it yet. The copy sets each of the task's fields by hand. A snapshot test lists the task's fields, so it fails if iceberg-rust adds a field the copy would leave out.
With iceberg.dynamic_filtering_enabled on, the scan now gives the reader the newest dynamic filters as it starts each file. The reader uses them to skip row groups, and to drop the rows that don't match. This helps joins on columns that aren't partition columns, and top-N. The reader checks every row once per IN list value. When the IN lists hold more than 200 values in total, we leave them out and keep the min/max bounds. The reader reads a column the file doesn't store as NULL. So a filter on an identity partition column, or on a column added later with a default value, can drop every row of such a file. We accept this for now. Any error leaves the file's task as it was.
Perseus
force-pushed
the
anirudh.singh/iceberg-dyn-filtering-3
branch
from
November 18, 2026 09:44
1b1af3d to
249fd05
Compare
Perseus
force-pushed
the
anirudh.singh/iceberg-dyn-filtering-4
branch
from
November 18, 2026 09:44
d5bec23 to
f0b5fb3
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
As the reader starts each file, the scan now ANDs the newest dynamic filters onto the file's own filter, so the reader can skip row groups the filters rule out. This covers joins on columns that aren't partition columns, and top-N.
The reader gets some columns wrong, so filters on them are left out: identity partition columns (the partition skipper checks those), columns with a default value, and nested columns. Above 200 listed values, IN lists are dropped and their bounds kept. The reader only gets the filters when row group filtering is on and page skipping is off, because page skipping gets widened columns wrong. Any error leaves the file as it is.