Repository navigation
fix(reader): page-index pruning drops rows for INT32/INT64 decimals - #3314
Merged
Merged
Conversation
Page-index bounds for INT32/INT64 columns were built as Int/Long literals regardless of the Iceberg field type. For decimal fields, query datums use Int128, and Datum::partial_cmp only compares decimals when both sides are Int128, so every comparison returned None. This silently pruned all matching pages for inequality and IN predicates when page-index row selection was enabled. Convert INT32/INT64 page bounds to Int128 for decimal fields, matching the row-group statistics path. Follow-up: FIXED_LEN_BYTE_ARRAY decimals (precision > 18) still get no page-index pruning, even though the row-group stats path decodes them. This only over-selects pages (never drops rows) and is pre-existing. Decoding them here later must guard against truncated ColumnIndex bounds (column_index_truncate_length). Closes apache#3264
Contributor
|
Thanks @anoopj |
dannycjones
pushed a commit
to dannycjones/iceberg-rust
that referenced
this pull request
Oct 9, 2026
…pache#3328) Decimals with precision > 18 are stored as FIXED_LEN_BYTE_ARRAY, which the page-index evaluator skipped, so inequality and IN predicates on them got no page pruning (over-selection only, never dropped rows). Decode those bounds with i128_from_be_bytes, matching the row-group statistics path. Guard against truncated ColumnIndex bounds: a bound whose width differs from the column's declared type_length signals a column_index_truncate_length truncation, and a truncated byte prefix no longer preserves the two's complement numeric ordering of decimals. Reject it so the column skips pruning rather than decode the wrong number. parquet-rs never truncates decimal column indexes, so this only defends against files written by other implementations. Benchmark A throwaway benchmark on a 1,000,000-row single-column decimal(30,2) file (precision > 18 -> FIXED_LEN_BYTE_ARRAY), 100 pages in one row group, read fully in memory so timing is decode-only. A `col > threshold` predicate sweeps selectivity. "read full file" is today's behavior (FLBA decimals skip page pruning); "read ms" applies the page-index RowSelection this change produces. 1,000,000 rows in 100 pages, 1 row group read full file : 3.431 ms predicate (keep) | rows read | eval us | read ms | speedup 1% | 10,000 | 17.5 | 0.038 | 90.0x 10% | 100,000 | 17.1 | 0.330 | 10.4x 50% | 500,000 | 16.6 | 1.629 | 2.1x 100% | 1,000,000 | 16.3 | 3.397 | 1.0x The win tracks predicate selectivity; eval() adds a constant ~17 us per row group per scan, negligible against the millisecond read. The benefit only materializes where page-index row selection runs: MoR tables or with_row_selection_enabled(true). Reproducer in the PR description. Follow-up to apache#3314.
This was referenced Oct 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #3264
What changes are included in this PR?
Page-index bounds for
INT32/INT64columns were built asInt/Longliterals regardless of the Iceberg field type. For decimal fields, query datums useInt128, and Datum::partial_cmp only compares decimals when both sides are Int128, so every comparison returned None. This silently pruned all matching pages for inequality and IN predicates when page-index row selection was enabled.Convert
INT32/INT64page bounds toInt128for decimal fields, matching the row-group statistics path.Follow-up: FIXED_LEN_BYTE_ARRAY decimals (precision > 18) still get no page-index pruning, even though the row-group stats path decodes them. This only over-selects pages (never drops rows) and is pre-existing. Decoding them here later must guard against truncated ColumnIndex bounds (column_index_truncate_length).
Are these changes tested?
Added new unit tests.