Repository navigation
Parquet: Prune nested structs to the projected subtree - #18336
Merged
Merged
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PruneColumns.struct()adds the original, unpruned Parquet field whenever the field's id is inselectedIds. A parent struct's id is inselectedIdswhenever any of its descendants is projected, so projecting one leaf of a deeply nested struct widens the read schema back to the full intermediate struct.This adds the pruned field instead. It also accumulates
hasChange(|=), so a fully projected sibling visited later cannot reset pruning recorded by an earlier sibling at the same level.Example: for
l1 struct<l2 struct<x, l3 struct<leaf, big>>>, projectingl1.l2.l3.leafpreviously read all ofl2. With this change it reads onlyl1.l2.l3.leaf. Whole-struct and top-level projections produce the same read schema as before. In our environment, reading a single deep leaf from a ~17 GB table took about 8x the scan task time of an id-only read, close to reading the whole parent struct.Notes for reviewers:
field == originalFieldat every level and the old path is kept.list()andmap()are unchanged. Structs inside list elements and map values prune correctly oncestruct()returns the pruned field.else if (field != null)branch now adds the pruned field instead of the original. This only matters for files where some fields have no ids. It should be more correct, but this is the area where review would help most.This revives #12634 and #14744, which were closed as stale, and addresses #11332.
Tests: 6 new tests in
TestPruneColumns. 4 fail onmainand pass with this change (testDeeplyNestedStructProjection,testDeeplyNestedStructMixed,testDeeplyNestedStructInsideList,testDeeplyNestedStructPartiallyProjectedBeforeFullyProjected). 2 pin unchanged behavior (testDeeplyNestedStructWhole,testNestedStructInsideMap)../gradlew :iceberg-parquet:checkpasses.AI Disclosure