Skip to content

Parquet: Fix nested field pruning in PruneColumns - #14744

Closed
IgorBerman wants to merge 1 commit into
apache:mainfrom
IgorBerman:fix/prune-columns-nested-fields
Closed

IgorBerman wants to merge 1 commit into
apache:mainfrom
IgorBerman:fix/prune-columns-nested-fields

Conversation

@IgorBerman

Copy link
Copy Markdown

When pruning nested structures (lists, maps, structs), the PruneColumns visitor was incorrectly returning the original unpruned field when the container's field ID was in the selectedIds set, even when child fields had been pruned.

This fix ensures that:

  1. In struct(): When a field is selected and has been pruned (field != originalField), use the pruned version instead of the original.
  2. In list(): Check for pruned element first before checking if elementId is selected, ensuring nested pruning is applied.
  3. In map(): Similarly check for pruned value before checking selected keys/values.
  4. Add validatePrunedField() to verify pruned fields maintain compatibility with original fields (same name, ID, and repetition).

This enables proper column pruning for deeply nested schemas like:
list<struct<field1, nested_list: list<struct<a, b, c, d>>>>

When projecting only field1 and nested_list[].a, b, the fix ensures fields c and d are properly pruned from the Parquet projection schema.

Integer fieldId = getId(originalField);
if (fieldId != null && selectedIds.contains(fieldId)) {
filteredFields.add(originalField);
// Use pruned field if available and different from original, otherwise use original

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This changes behavior and I don't think that it is correct because the field itself was selected. If a field is specifically selected by ID, then the original is used because it is completely selected.

This covers cases like SELECT struct_field, struct_field.sub_field FROM .... The sub-field is going to cause field here to be a projection of the struct, but the struct is also independently selected and so all of its fields should be returned and not pruned.

@rdblue rdblue left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the original behavior is correct.

@IgorBerman

Copy link
Copy Markdown
Author

@rdblue thanks for the review!
The question here what projection schema will look like in a case of SELECT struct_field, struct_field.sub_field FROM
I've added test that shows that if we use projection with full struct schema then the change will preserve struct_field fully(point in my test)
if it will contain only struct_field itself without all the inner fields - then probably it wont work.
this depends on how projection is created and defined probably
it comes from expectedSchema most probably and the question is how it's defined
this.projection = ParquetSchemaUtil.pruneColumns(fileSchema, expectedSchema);
Maybe we here entering domain of how spark creates expectedSchema. I'd argue that it should contain full struct_field schema.
I'll try to do more tests

@IgorBerman
IgorBerman force-pushed the fix/prune-columns-nested-fields branch from 65f1a4f to 80f0447 Compare December 4, 2025 06:29
When pruning nested structures (lists, maps, structs), the PruneColumns
visitor was incorrectly returning the original unpruned field when the
container's field ID was in the selectedIds set, even when child fields
had been pruned.

This fix ensures that:
1. In struct(): When a field is selected and has been pruned (field != originalField),
   use the pruned version instead of the original.
2. In list(): Check for pruned element first before checking if elementId
   is selected, ensuring nested pruning is applied.
3. In map(): Similarly check for pruned value before checking selected keys/values.
4. Add validatePrunedField() to verify pruned fields maintain compatibility
   with original fields (same name, ID, and repetition).

This enables proper column pruning for deeply nested schemas like:
list<struct<field1, nested_list: list<struct<a, b, c, d>>>>

When projecting only field1 and nested_list[].a, b, the fix ensures
fields c and d are properly pruned from the Parquet projection schema.

Note: When a struct is explicitly selected (SELECT struct_field, struct_field.sub_field),
the full struct is preserved because field == originalField in that case.
@IgorBerman
IgorBerman force-pushed the fix/prune-columns-nested-fields branch from 80f0447 to 610fd12 Compare December 4, 2025 06:50
@IgorBerman

Copy link
Copy Markdown
Author

let's continue conversation in slack meanwhile https://apache-iceberg.slack.com/archives/C03LG1D563F/p1764683146862199

@github-actions

github-actions Bot commented Jan 4, 2026

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Jan 4, 2026
@github-actions

Copy link
Copy Markdown

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants