Repository navigation
perf(reader): read leaf field IDs from metadata in the projection mask - #3396
Conversation
get_arrow_projection_mask_with_field_ids carried each leaf's field ID from its first pass to its second through a HashMap<FieldRef, i32>. Hashing an Arrow Field hashes its whole DataType and allocates a sorted copy of its metadata keys, and the map grew from empty. Read the ID from the leaf's PARQUET:field_id metadata in both passes instead. Closes apache#3394
There was a problem hiding this comment.
🟢 Approval recommended
The focused refactor preserves existing projection behavior while eliminating unnecessary hashing and allocations.
0 open findings
What changed in this PR
Optimizes projection-mask construction by reading Parquet field IDs directly from Arrow metadata, resolving #3394 without changing behavior.
Changes:
- Removes the costly
FieldRef-keyed intermediary map. - Reuses a metadata parsing helper across projection and field-ID mapping.
| File | Description |
|---|---|
crates/iceberg/src/arrow/reader/projection.rs |
Streamlines field-ID lookup during projection. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
laskoviymishka
left a comment
There was a problem hiding this comment.
Nice and clean refactoring!
| projected_fields.insert((*f).clone(), field_id); | ||
| leaf_field_id_set.contains(&field_id) | ||
| }) | ||
| parquet_field_id(f).is_some_and(|field_id| leaf_field_id_set.contains(&field_id)) |
There was a problem hiding this comment.
tiny thing — the second pass and the build_field_id_map_from_arrow_schema call site both name this field; only this first-pass closure calls it f. I'd match them so it reads consistently.
| column_map | ||
| } | ||
|
|
||
| /// Returns the field ID in `field`'s `PARQUET:field_id` metadata, if it has a valid one. |
There was a problem hiding this comment.
I'd spell out what None means here — key absent or value not a valid i32 — since that case isn't a no-op: the leaf drops out of the projection and gets null-filled downstream, so a corrupt PARQUET:field_id surfaces as a missing column rather than an error. One line does it:
| /// Returns the field ID in `field`'s `PARQUET:field_id` metadata, if it has a valid one. | |
| /// Returns the field ID stored in `field`'s `PARQUET:field_id` metadata, or `None` if the key is absent or not a valid `i32`. |
| } | ||
|
|
||
| /// Returns the field ID in `field`'s `PARQUET:field_id` metadata, if it has a valid one. | ||
| fn parquet_field_id(field: &Field) -> Option<i32> { |
There was a problem hiding this comment.
The whole equivalence rests on None → excluded in the second pass, and that's the one branch the existing tests don't pin. A small unit test would lock it: one leaf with a valid id, one with no metadata, one with "abc", asserting only the valid leaf lands in the mask. Not blocking — the happy path is well covered — but it'd guard build_field_id_map_from_arrow_schema against the same edge too.
Which issue does this PR close?
What changes are included in this PR?
get_arrow_projection_mask_with_field_idscarried each leaf's field ID from its first pass over the Arrow leaves to the second through aHashMap<FieldRef, i32>. Hashing an ArrowFieldhashes its wholeDataTypeand allocates a sorted copy of its metadata keys, and the map grew from empty. The second pass now reads the ID from the leaf'sPARQUET:field_idmetadata, as the first pass already does, and the map is gone.build_field_id_map_from_arrow_schemauses the same small helper. Behavior does not change.One call to
ArrowReader::get_arrow_projection_maskfor a file whose columns are allintwith field IDs (release build, Apple M3 Max, 500 calls each, two runs):The numbers come from the local counting-allocator test in #3394, which is not part of this PR.
This conflicts with #3384 on one line, the declaration of the map, which #3384 sizes up front and this PR removes.
Are these changes tested?
Existing tests cover the projection mask, such as
test_arrow_projection_maskand the reader tests inprojection.rs.cargo test -p iceberg --libpasses, and clippy is clean.AI Disclosure
Drafted with Claude Code. The numbers come from the test in #3394, run against
mainatb2e90754band this branch.