Skip to content

perf(reader): read leaf field IDs from metadata in the projection mask - #3396

Merged
laskoviymishka merged 1 commit into
apache:mainfrom
comphead:perf-mask-field-id-lookup
Oct 11, 2026
Merged

laskoviymishka merged 1 commit into
apache:mainfrom
comphead:perf-mask-field-id-lookup

Conversation

@comphead

Copy link
Copy Markdown
Collaborator

Which issue does this PR close?

What changes are included in this PR?

get_arrow_projection_mask_with_field_ids carried each leaf's field ID from its first pass over the Arrow leaves to the second through a HashMap<FieldRef, i32>. Hashing an Arrow Field hashes its whole DataType and allocates a sorted copy of its metadata keys, and the map grew from empty. The second pass now reads the ID from the leaf's PARQUET:field_id metadata, as the first pass already does, and the map is gone. build_field_id_map_from_arrow_schema uses the same small helper. Behavior does not change.

One call to ArrowReader::get_arrow_projection_mask for a file whose columns are all int with field IDs (release build, Apple M3 Max, 500 calls each, two runs):

Columns Projected Allocations before Allocations after Time before Time after
100 10 486 172 34 to 46 µs 13 µs
1,000 10 3,976 178 318 µs 95 µs
1,000 1,000 15,915 12,117 897 µs 657 to 660 µs

The numbers come from the local counting-allocator test in #3394, which is not part of this PR.

This conflicts with #3384 on one line, the declaration of the map, which #3384 sizes up front and this PR removes.

Are these changes tested?

Existing tests cover the projection mask, such as test_arrow_projection_mask and the reader tests in projection.rs. cargo test -p iceberg --lib passes, and clippy is clean.

AI Disclosure

Drafted with Claude Code. The numbers come from the test in #3394, run against main at b2e90754b and this branch.

get_arrow_projection_mask_with_field_ids carried each leaf's field ID
from its first pass to its second through a HashMap<FieldRef, i32>.
Hashing an Arrow Field hashes its whole DataType and allocates a sorted
copy of its metadata keys, and the map grew from empty. Read the ID
from the leaf's PARQUET:field_id metadata in both passes instead.

Closes apache#3394
@comphead
comphead marked this pull request as ready for review October 10, 2026 21:16
Copilot AI balanced review requested due to automatic review settings October 10, 2026 21:16

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused refactor preserves existing projection behavior while eliminating unnecessary hashing and allocations.

0 open findings

What changed in this PR

Optimizes projection-mask construction by reading Parquet field IDs directly from Arrow metadata, resolving #3394 without changing behavior.

Changes:

  • Removes the costly FieldRef-keyed intermediary map.
  • Reuses a metadata parsing helper across projection and field-ID mapping.
File Description
crates/​iceberg/​src/​arrow/​reader/​projection.rs Streamlines field-ID lookup during projection.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@laskoviymishka laskoviymishka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice and clean refactoring!

projected_fields.insert((*f).clone(), field_id);
leaf_field_id_set.contains(&field_id)
})
parquet_field_id(f).is_some_and(|field_id| leaf_field_id_set.contains(&field_id))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tiny thing — the second pass and the build_field_id_map_from_arrow_schema call site both name this field; only this first-pass closure calls it f. I'd match them so it reads consistently.

column_map
}

/// Returns the field ID in `field`'s `PARQUET:field_id` metadata, if it has a valid one.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd spell out what None means here — key absent or value not a valid i32 — since that case isn't a no-op: the leaf drops out of the projection and gets null-filled downstream, so a corrupt PARQUET:field_id surfaces as a missing column rather than an error. One line does it:

Suggested change
/// Returns the field ID in `field`'s `PARQUET:field_id` metadata, if it has a valid one.
/// Returns the field ID stored in `field`'s `PARQUET:field_id` metadata, or `None` if the key is absent or not a valid `i32`.

}

/// Returns the field ID in `field`'s `PARQUET:field_id` metadata, if it has a valid one.
fn parquet_field_id(field: &Field) -> Option<i32> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The whole equivalence rests on None → excluded in the second pass, and that's the one branch the existing tests don't pin. A small unit test would lock it: one leaf with a valid id, one with no metadata, one with "abc", asserting only the valid leaf lands in the mask. Not blocking — the happy path is well covered — but it'd guard build_field_id_map_from_arrow_schema against the same edge too.

@laskoviymishka
laskoviymishka added this pull request to the merge queue Oct 11, 2026
Merged via the queue into apache:main with commit d3e9f45 Oct 11, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(reader): projection mask hashes every Arrow leaf twice through a FieldRef map

3 participants