Repository navigation
fix column prunning 1.6.x - #14737
Closed
IgorBerman wants to merge 1 commit into
Closed
IgorBerman wants to merge 1 commit into
IgorBerman wants to merge 1 commit into
Conversation
When pruning nested structures (lists, maps, structs), the PruneColumns visitor was incorrectly returning the original unpruned field when the container's field ID was in the selectedIds set, even when child fields had been pruned. This fix ensures that: 1. In struct(): When a field is selected and has been pruned (field != originalField), use the pruned version instead of the original. 2. In list(): Check for pruned element first before checking if elementId is selected, ensuring nested pruning is applied. 3. In map(): Similarly check for pruned value before checking selected keys/values. 4. Add validatePrunedField() to verify pruned fields maintain compatibility with original fields (same name, ID, and repetition). This enables proper column pruning for deeply nested schemas like: list<struct<field1, nested_list: list<struct<a, b, c, d>>>> When projecting only field1 and nested_list[].a, b, the fix ensures fields c and d are properly pruned from the Parquet projection schema.
IgorBerman
force-pushed
the
feature/SPARK-47230-taboola-schema-on-read-as-part-of-spark
branch
from
December 3, 2025 09:32
8543c0d to
338413e
Compare
Author
|
closing in favor to #12634 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When pruning nested structures (lists, maps, structs), the PruneColumns
visitor was incorrectly returning the original unpruned field when the
container's field ID was in the selectedIds set, even when child fields
had been pruned.
This fix ensures that:
use the pruned version instead of the original.
is selected, ensuring nested pruning is applied.
with original fields (same name, ID, and repetition).
This enables proper column pruning for deeply nested schemas like:
list<struct<field1, nested_list: list<struct<a, b, c, d>>>>
When projecting only field1 and nested_list[].a, b, the fix ensures
fields c and d are properly pruned from the Parquet projection schema.