Skip to content

fix(reader): page-index pruning drops rows for INT32/INT64 decimals - #3314

Merged
Fokko merged 2 commits into
apache:mainfrom
anoopj:fix-3264-page-index-decimal
Oct 2, 2026
Merged

Fokko merged 2 commits into
apache:mainfrom
anoopj:fix-3264-page-index-decimal

Conversation

@anoopj

@anoopj anoopj commented Sep 30, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #3264

What changes are included in this PR?

Page-index bounds for INT32/INT64 columns were built as Int/Long literals regardless of the Iceberg field type. For decimal fields, query datums use Int128, and Datum::partial_cmp only compares decimals when both sides are Int128, so every comparison returned None. This silently pruned all matching pages for inequality and IN predicates when page-index row selection was enabled.

Convert INT32/INT64 page bounds to Int128 for decimal fields, matching the row-group statistics path.

Follow-up: FIXED_LEN_BYTE_ARRAY decimals (precision > 18) still get no page-index pruning, even though the row-group stats path decodes them. This only over-selects pages (never drops rows) and is pre-existing. Decoding them here later must guard against truncated ColumnIndex bounds (column_index_truncate_length).

Are these changes tested?

Added new unit tests.

anoopj and others added 2 commits September 30, 2026 15:19
Page-index bounds for INT32/INT64 columns were built as Int/Long literals
regardless of the Iceberg field type. For decimal fields, query datums use
Int128, and Datum::partial_cmp only compares decimals when both sides are
Int128, so every comparison returned None. This silently pruned all matching
pages for inequality and IN predicates when page-index row selection was
enabled.

Convert INT32/INT64 page bounds to Int128 for decimal fields, matching the
row-group statistics path.

Follow-up: FIXED_LEN_BYTE_ARRAY decimals (precision > 18) still get no
page-index pruning, even though the row-group stats path decodes them. This
only over-selects pages (never drops rows) and is pre-existing. Decoding them
here later must guard against truncated ColumnIndex bounds
(column_index_truncate_length).

Closes apache#3264

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ooh, this is a big one since it can drop data. Thanks for catching and fixing this @anoopj

@Fokko
Fokko added this pull request to the merge queue Oct 2, 2026
@Fokko

Fokko commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Thanks @anoopj

Merged via the queue into apache:main with commit ffa864b Oct 2, 2026
24 checks passed
@anoopj
anoopj deleted the fix-3264-page-index-decimal branch October 2, 2026 15:17
dannycjones pushed a commit to dannycjones/iceberg-rust that referenced this pull request Oct 9, 2026
…pache#3328)

Decimals with precision > 18 are stored as FIXED_LEN_BYTE_ARRAY, which the
page-index evaluator skipped, so inequality and IN predicates on them got no
page pruning (over-selection only, never dropped rows). Decode those bounds
with i128_from_be_bytes, matching the row-group statistics path.

Guard against truncated ColumnIndex bounds: a bound whose width differs from
the column's declared type_length signals a column_index_truncate_length
truncation, and a truncated byte prefix no longer preserves the two's
complement numeric ordering of decimals. Reject it so the column skips pruning
rather than decode the wrong number. parquet-rs never truncates decimal column
indexes, so this only defends against files written by other implementations.

Benchmark

A throwaway benchmark on a 1,000,000-row single-column decimal(30,2) file
(precision > 18 -> FIXED_LEN_BYTE_ARRAY), 100 pages in one row group, read
fully in memory so timing is decode-only. A `col > threshold` predicate sweeps
selectivity. "read full file" is today's behavior (FLBA decimals skip page
pruning); "read ms" applies the page-index RowSelection this change produces.

    1,000,000 rows in 100 pages, 1 row group
    read full file : 3.431 ms

      predicate (keep) | rows read | eval us | read ms | speedup
                   1%  |    10,000 |    17.5 |   0.038 |   90.0x
                  10%  |   100,000 |    17.1 |   0.330 |   10.4x
                  50%  |   500,000 |    16.6 |   1.629 |    2.1x
                 100%  | 1,000,000 |    16.3 |   3.397 |    1.0x

The win tracks predicate selectivity; eval() adds a constant ~17 us per row
group per scan, negligible against the millisecond read. The benefit only
materializes where page-index row selection runs: MoR tables or
with_row_selection_enabled(true). Reproducer in the PR description.

Follow-up to apache#3314.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Page-index pruning can drop rows for INT32/INT64-encoded decimals

2 participants