Skip to content

Parquet: Prune nested structs to the projected subtree - #18336

Merged
bryanck merged 4 commits into
apache:mainfrom
jzhuge:parquet-prune-nested-structs
Oct 12, 2026
Merged

bryanck merged 4 commits into
apache:mainfrom
jzhuge:parquet-prune-nested-structs

Conversation

@jzhuge

@jzhuge jzhuge commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

PruneColumns.struct() adds the original, unpruned Parquet field whenever the field's id is in selectedIds. A parent struct's id is in selectedIds whenever any of its descendants is projected, so projecting one leaf of a deeply nested struct widens the read schema back to the full intermediate struct.

This adds the pruned field instead. It also accumulates hasChange (|=), so a fully projected sibling visited later cannot reset pruning recorded by an earlier sibling at the same level.

Example: for l1 struct<l2 struct<x, l3 struct<leaf, big>>>, projecting l1.l2.l3.leaf previously read all of l2. With this change it reads only l1.l2.l3.leaf. Whole-struct and top-level projections produce the same read schema as before. In our environment, reading a single deep leaf from a ~17 GB table took about 8x the scan task time of an id-only read, close to reading the whole parent struct.

Notes for reviewers:

  • A selected struct is still read in full. Projections expand a selected struct to its full subtree, so field == originalField at every level and the old path is kept.
  • list() and map() are unchanged. Structs inside list elements and map values prune correctly once struct() returns the pruned field.
  • The else if (field != null) branch now adds the pruned field instead of the original. This only matters for files where some fields have no ids. It should be more correct, but this is the area where review would help most.

This revives #12634 and #14744, which were closed as stale, and addresses #11332.

Tests: 6 new tests in TestPruneColumns. 4 fail on main and pass with this change (testDeeplyNestedStructProjection, testDeeplyNestedStructMixed, testDeeplyNestedStructInsideList, testDeeplyNestedStructPartiallyProjectedBeforeFullyProjected). 2 pin unchanged behavior (testDeeplyNestedStructWhole, testNestedStructInsideMap). ./gradlew :iceberg-parquet:check passes.


AI Disclosure

  • Model: Claude Opus 5.5
  • Platform/Tool: Claude Code
  • Human Oversight: fully reviewed
  • Prompt Summary: Prune Parquet columns to the projected subtree for nested structs, with regression tests.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants