Repository navigation
Parquet: Fix nested field pruning in PruneColumns - #14744
IgorBerman wants to merge 1 commit into
Conversation
| Integer fieldId = getId(originalField); | ||
| if (fieldId != null && selectedIds.contains(fieldId)) { | ||
| filteredFields.add(originalField); | ||
| // Use pruned field if available and different from original, otherwise use original |
There was a problem hiding this comment.
This changes behavior and I don't think that it is correct because the field itself was selected. If a field is specifically selected by ID, then the original is used because it is completely selected.
This covers cases like SELECT struct_field, struct_field.sub_field FROM .... The sub-field is going to cause field here to be a projection of the struct, but the struct is also independently selected and so all of its fields should be returned and not pruned.
rdblue
left a comment
There was a problem hiding this comment.
I think the original behavior is correct.
|
@rdblue thanks for the review! |
65f1a4f to
80f0447
Compare
When pruning nested structures (lists, maps, structs), the PruneColumns visitor was incorrectly returning the original unpruned field when the container's field ID was in the selectedIds set, even when child fields had been pruned. This fix ensures that: 1. In struct(): When a field is selected and has been pruned (field != originalField), use the pruned version instead of the original. 2. In list(): Check for pruned element first before checking if elementId is selected, ensuring nested pruning is applied. 3. In map(): Similarly check for pruned value before checking selected keys/values. 4. Add validatePrunedField() to verify pruned fields maintain compatibility with original fields (same name, ID, and repetition). This enables proper column pruning for deeply nested schemas like: list<struct<field1, nested_list: list<struct<a, b, c, d>>>> When projecting only field1 and nested_list[].a, b, the fix ensures fields c and d are properly pruned from the Parquet projection schema. Note: When a struct is explicitly selected (SELECT struct_field, struct_field.sub_field), the full struct is preserved because field == originalField in that case.
80f0447 to
610fd12
Compare
|
let's continue conversation in slack meanwhile https://apache-iceberg.slack.com/archives/C03LG1D563F/p1764683146862199 |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
When pruning nested structures (lists, maps, structs), the PruneColumns visitor was incorrectly returning the original unpruned field when the container's field ID was in the selectedIds set, even when child fields had been pruned.
This fix ensures that:
This enables proper column pruning for deeply nested schemas like:
list<struct<field1, nested_list: list<struct<a, b, c, d>>>>
When projecting only field1 and nested_list[].a, b, the fix ensures fields c and d are properly pruned from the Parquet projection schema.