Skip to content

fix column prunning 1.6.x - #14737

Closed
IgorBerman wants to merge 1 commit into
apache:1.6.xfrom
IgorBerman:feature/SPARK-47230-taboola-schema-on-read-as-part-of-spark
Closed

IgorBerman wants to merge 1 commit into
apache:1.6.xfrom
IgorBerman:feature/SPARK-47230-taboola-schema-on-read-as-part-of-spark

Conversation

@IgorBerman

@IgorBerman IgorBerman commented Dec 2, 2025 •

Copy link
Copy Markdown

When pruning nested structures (lists, maps, structs), the PruneColumns
visitor was incorrectly returning the original unpruned field when the
container's field ID was in the selectedIds set, even when child fields
had been pruned.

This fix ensures that:

  1. In struct(): When a field is selected and has been pruned (field != originalField),
    use the pruned version instead of the original.
  2. In list(): Check for pruned element first before checking if elementId
    is selected, ensuring nested pruning is applied.
  3. In map(): Similarly check for pruned value before checking selected keys/values.
  4. Add validatePrunedField() to verify pruned fields maintain compatibility
    with original fields (same name, ID, and repetition).

This enables proper column pruning for deeply nested schemas like:
list<struct<field1, nested_list: list<struct<a, b, c, d>>>>

When projecting only field1 and nested_list[].a, b, the fix ensures
fields c and d are properly pruned from the Parquet projection schema.

@IgorBerman
IgorBerman changed the base branch from main to 1.6.x December 2, 2025 15:34
When pruning nested structures (lists, maps, structs), the PruneColumns
visitor was incorrectly returning the original unpruned field when the
container's field ID was in the selectedIds set, even when child fields
had been pruned.

This fix ensures that:
1. In struct(): When a field is selected and has been pruned (field != originalField),
   use the pruned version instead of the original.
2. In list(): Check for pruned element first before checking if elementId
   is selected, ensuring nested pruning is applied.
3. In map(): Similarly check for pruned value before checking selected keys/values.
4. Add validatePrunedField() to verify pruned fields maintain compatibility
   with original fields (same name, ID, and repetition).

This enables proper column pruning for deeply nested schemas like:
list<struct<field1, nested_list: list<struct<a, b, c, d>>>>

When projecting only field1 and nested_list[].a, b, the fix ensures
fields c and d are properly pruned from the Parquet projection schema.
@IgorBerman
IgorBerman force-pushed the feature/SPARK-47230-taboola-schema-on-read-as-part-of-spark branch from 8543c0d to 338413e Compare December 3, 2025 09:32
@IgorBerman IgorBerman changed the title Feature/spark 47230 taboola schema on read as part of spark fix column prunning 1.6.x Dec 3, 2025
@IgorBerman IgorBerman closed this Dec 7, 2025
@IgorBerman

Copy link
Copy Markdown
Author

closing in favor to #12634

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant