Skip to content

Parquet: Fix column pruning for deeply nested fields - #12634

Closed
sriharshaj wants to merge 3 commits into
apache:mainfrom
sriharshaj:fix-parquet-column-pruning
Closed

sriharshaj wants to merge 3 commits into
apache:mainfrom
sriharshaj:fix-parquet-column-pruning

Conversation

@sriharshaj

@sriharshaj sriharshaj commented Mar 25, 2025 •

Copy link
Copy Markdown

This PR addresses a performance issue related to column pruning in nested structures when reading Iceberg tables from Parquet files.

Issue

In the current implementation of PruneColumns#struct, we are including the entire parent column in the read schema, even when only a subset of nested fields is required. This occurs because:

The originalField is being added to the filteredFields regardless of whether the passed filtered field is already present. This behavior causes the entire parent column to be read if any leaf node within the nested structure is selected in a query.

Proposed solution

I updated the code to use passed filtered field when present, and use originalField when passed filtered field is null.

@sriharshaj
sriharshaj marked this pull request as ready for review March 25, 2025 02:27
@sriharshaj

Copy link
Copy Markdown
Author

I am not sure whether the use of originalField even when field is present is intentional behavior or a bug.

@sriharshaj

sriharshaj commented Mar 25, 2025 •

Copy link
Copy Markdown
Author

Following can be the issues related to this:
#12428
#11617

@sriharshaj
sriharshaj force-pushed the fix-parquet-column-pruning branch from 213bf7b to 69dfcd9 Compare March 26, 2025 16:13
@amogh-jahagirdar
amogh-jahagirdar self-requested a review March 26, 2025 17:21
@github-actions

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Apr 26, 2025
@sriharshaj

Copy link
Copy Markdown
Author

@amogh-jahagirdar Can you please take a look? Thanks.

@github-actions github-actions Bot removed the stale label Apr 29, 2025
@github-actions

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label May 29, 2025
@sriharshaj

Copy link
Copy Markdown
Author

Can someone please take a look?

@github-actions github-actions Bot removed the stale label Jun 3, 2025
@github-actions

github-actions Bot commented Jul 3, 2025

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Jul 3, 2025
@sriharshaj

Copy link
Copy Markdown
Author

Commenting to keep it alive.

@github-actions github-actions Bot removed the stale label Jul 4, 2025
@github-actions

github-actions Bot commented Aug 3, 2025

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Aug 3, 2025
@sriharshaj

Copy link
Copy Markdown
Author

Commenting to keep it alive.

@github-actions github-actions Bot removed the stale label Aug 6, 2025

@amogh-jahagirdar amogh-jahagirdar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK my apologies @sriharshaj for the delay on reviewing this, I got a chance today to step through this and I do believe that this fix is correct. I would like to get @rdblue take on this if he gets a chance but just had a minor comment in the test.

Comment on lines -93 to -94
if (fieldId != null && selectedIds.contains(fieldId)) {
filteredFields.add(originalField);

@amogh-jahagirdar amogh-jahagirdar Aug 20, 2025 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My thoughts on why this is correct:

Prior to this PR, the check being performed first prevented column pruning on deeply nested fields because selectedIds contains the IDs of structs along with any further nested fields being projected. This check would pass first and we would just add the whole originalField which includes fields that may not be needed to read to satisfy a read.

In the change, by checking field is not null first we're checking the pruned fields of the struct since this visitor works in a preorder fashion where a structure is visited and then those contents are passed here in that List. If a pruned field exists then we add that field (and not the original field since that would not be pruned).

OK I think I'm convinced that this change is correct but I'd like @rdblue take on this.

if (field != null) {
filteredFields.add(field);
hasChange = true;
} else if (fieldId != null && selectedIds.contains(fieldId)) {

@rdblue rdblue Aug 20, 2025 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that the order should change. If the field itself is selected by ID then the entire field should be projected. For instance, you could have a case where multiple fields are referenced:

SELECT response FROM http_log WHERE response.status = 200

That references response.status specifically, but the entire response struct should be produced.

I think I do agree with the secondary case where if the field was returned from traversing the schema, then at least one sub-field was projected and the projected version should be returned.

@sriharshaj sriharshaj Aug 20, 2025 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we select the entire struct column like:

SELECT response FROM http_log

response is expanded into all of its nested fields, and adds every nested field’s id to the selected id list.

By the time we reach this code path, field already matches originalField.

So in the above case you mentioned, it will return correct projection.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right @sriharshaj,

so @rdblue given

SELECT response FROM http_log WHERE response.status = 200

selectedIds ends up including the ID from the top level struct and all of it's nested fields. As a result, I think we do first want to check anything from the pruned set of fields. If that's defined AND the whole struct is projected that ends up working anyways because the set of fields for the struct already includes everything from the preorder traversal (i.e. the pruning didn't prune anything since there's nothing to prune from response).

This relies on the fact that selectedIds includes the fields from the top level struct and it's nested fields. I think this is fine because the only interface to PruneColumns is ParquetSchemaUtils.pruneColumns which always includes all these fields anyways in the implementation.

I also wrote a quick test (albeit via Spark) to prove it but this logic is all embedded in our parquet APIs anyways so it applies to any integration using those.

  @TestTemplate
  public void testStructProjection() {
    assumeThat(fileFormat).isEqualTo(FileFormat.PARQUET);
    sql("CREATE TABLE %s (response struct<code:int, body:string>) using iceberg", tableName);
    sql("insert into %s values named_struct(\"code\", 200, \"body\", \"success\")", tableName);
    sql("insert into %s values named_struct(\"code\", 503, \"body\", \"because istio\")", tableName);
    List<Object[]> results = sql("SELECT response from %s where response.code > 500", tableName);

    assertThat(results).hasSize(1);
    assertThat(results.get(0)[0]).isEqualTo(new Object[]{503, "because istio"});
  }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was re-reading my post, I meant post-order not pre-order (since the child fields are visited and pruned first) but my point still stands I think.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you're right that the current way that this class is called by ParquetSchemaUtil.pruneColumns will always pass the struct IDs in. Here's that method:

  public static MessageType pruneColumns(MessageType fileSchema, Schema expectedSchema) {
    // column order must match the incoming type, so it doesn't matter that the ids are unordered
    Set<Integer> selectedIds = TypeUtil.getProjectedIds(expectedSchema);
    return (MessageType)
        TypeWithSchemaVisitor.visit(
            expectedSchema.asStruct(), fileSchema, new PruneColumns(selectedIds));
  }

The call to TypeUtil.getProjectedIds calls getIdsInternal(schema.asStruct(), true /*includeStructIds*/). If we keep the previous order of the if blocks as I suggested, then this call would always produce the original struct, which doesn't fix column pruning. However, I think we do want to keep the original order of the blocks and fix the problem by setting includeStructIds to false. That way we don't incorrectly select entire structs unless they are explicitly selected.

I think that the behavior of selecting the entire struct when its field ID is passed in is correct, for the reason I gave in my comment above. I don't think that we should change the behavior of the visitor just because we know how it is called. That creates a dependency for correct behavior on how it is called, which we don't want to assume. We could add other uses of this visitor in the future and it should still behave predictably if we reuse it and don't make the same choices in how it is called.

@sriharshaj sriharshaj Sep 16, 2025 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rdblue
From my reading of the code, setting includeStructIds to false doesn’t consistently add the struct field ID to selectedIds. As a result, (fieldId != null && selectedIds.contains(fieldId)) may never evaluate to true.

Also, given the current implementation, when we only look at expectedSchema in:

TypeUtil.getProjectedIds(expectedSchema);

it seems we can’t determine whether a subset of struct fields is selected or the entire struct is selected, since there’s no table schema to diff against.

In addition, even if only one nested field of a struct is selected, it looks like org.apache.iceberg.Schema can’t be built without the parent struct field ID.

One possible way to distinguish "entire struct selected" from "subset selected" would be to omit the struct’s nested fields from the Schema when the full struct is selected (In this case above change should work).

Please let me know if I am missing anything.

@sriharshaj
sriharshaj requested a review from rdblue August 26, 2025 18:00
@github-actions

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Oct 17, 2025
@sriharshaj

Copy link
Copy Markdown
Author

@amogh-jahagirdar and @rdblue Can you take a look at PR again? Thanks.

@github-actions github-actions Bot removed the stale label Oct 22, 2025
@IgorBerman

Copy link
Copy Markdown

@sriharshaj Hi,
if I may ask question of understanding: I know that there is large code base in spark itself for column pruning, and now you are changing internal to iceberg functionality that is related to same area.
What principle defines separation between two(spark vs iceberg)? How have you decided that you need to improve pruning of columns in iceberg itself and not in spark(e.g.)?
ps: I'm trying to work on supporting pruning of nested fields inside arrays of structs and personally I'm focusing on spark codebase.

@sriharshaj

Copy link
Copy Markdown
Author

@sriharshaj Hi, if I may ask question of understanding: I know that there is large code base in spark itself for column pruning, and now you are changing internal to iceberg functionality that is related to same area. What principle defines separation between two(spark vs iceberg)? How have you decided that you need to improve pruning of columns in iceberg itself and not in spark(e.g.)? ps: I'm trying to work on supporting pruning of nested fields inside arrays of structs and personally I'm focusing on spark codebase.

Hey @IgorBerman,
My understanding is that based on the Spark query or DF logic, Iceberg decides which columns need to be read from the Parquet data. So, I believe the change should happen at the Iceberg level. WDYT?

@IgorBerman

Copy link
Copy Markdown

I was checking this mechanism lately and I think there are 2 parts in the story.
First, Spark computes "needed" schema and can prune schema by different optimization rules. Then it passes this schema to Iceberg level and then there is additional PruneColumns logic that Iceberg doing, in which Iceberg trying to take file schema and needed schema(computed by spark) and decide what columns from parquet(e.g.) to read(by judging if needed schema is sound with file schema etc). I.e. IO pruning depends on the later logic, but it takes into account what Spark computed

@sriharshaj

Copy link
Copy Markdown
Author

I was checking this mechanism lately and I think there are 2 parts in the story. First, Spark computes "needed" schema and can prune schema by different optimization rules. Then it passes this schema to Iceberg level and then there is additional PruneColumns logic that Iceberg doing, in which Iceberg trying to take file schema and needed schema(computed by spark) and decide what columns from parquet(e.g.) to read(by judging if needed schema is sound with file schema etc). I.e. IO pruning depends on the later logic, but it takes into account what Spark computed

Yeah, that sounds right. Optimizations on the Spark side may not be applied when reading the data from Parquet.

@sriharshaj
sriharshaj force-pushed the fix-parquet-column-pruning branch from ef071bc to 2865afc Compare December 5, 2025 21:49
@IgorBerman

Copy link
Copy Markdown

ok, I was a bit blind to what you are doing and did something similar(after you opened your PR and actually I commented on it before) due to our usecase (where we want pruning of columns inside structs inside arrays #14744). I needed to support cases of pruning not only due to structs chains but also for struct<array<struct<array>> cases
I'll try to test your fix with my tests to see if it gives same results

@IgorBerman

Copy link
Copy Markdown

ok, verified and your fix passes also my tests containing nested arrays of structs
can you please take few tests from https://github.com/apache/iceberg/pull/14744/files#diff-1bbe7597b597eead2808831e8b9c9ac16dc8a2a912670cc8a78f5b09584b4468
then I'll close my PR as duplicate

@sriharshaj

sriharshaj commented Dec 8, 2025 •

Copy link
Copy Markdown
Author

ok, verified and your fix passes also my tests containing nested arrays of structs
can you please take few tests from https://github.com/apache/iceberg/pull/14744/files#diff-1bbe7597b597eead2808831e8b9c9ac16dc8a2a912670cc8a78f5b09584b4468
then I'll close my PR as duplicate

Thanks for testing. Sure, I will add more of them.

@github-actions

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Jan 10, 2026
@github-actions

Copy link
Copy Markdown

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

@github-actions github-actions Bot closed this Jan 18, 2026
jzhuge added a commit to jzhuge/iceberg that referenced this pull request Oct 6, 2026
Address review feedback: document in PruneColumns and
ParquetSchemaUtil.pruneColumns that a partially projected struct is
intentional and kept to its projected subtree, resolving the concern
raised in apache#12634, and rename the map-value test whose name overstated
its nesting depth.

Generated-by: Claude Code (Claude Opus 5.5)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants