Repository navigation
Parquet: Fix column pruning for deeply nested fields - #12634
sriharshaj wants to merge 3 commits into
Conversation
|
I am not sure whether the use of |
213bf7b to
69dfcd9
Compare
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
@amogh-jahagirdar Can you please take a look? Thanks. |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
Can someone please take a look? |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
Commenting to keep it alive. |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
Commenting to keep it alive. |
amogh-jahagirdar
left a comment
There was a problem hiding this comment.
OK my apologies @sriharshaj for the delay on reviewing this, I got a chance today to step through this and I do believe that this fix is correct. I would like to get @rdblue take on this if he gets a chance but just had a minor comment in the test.
| if (fieldId != null && selectedIds.contains(fieldId)) { | ||
| filteredFields.add(originalField); |
There was a problem hiding this comment.
My thoughts on why this is correct:
Prior to this PR, the check being performed first prevented column pruning on deeply nested fields because selectedIds contains the IDs of structs along with any further nested fields being projected. This check would pass first and we would just add the whole originalField which includes fields that may not be needed to read to satisfy a read.
In the change, by checking field is not null first we're checking the pruned fields of the struct since this visitor works in a preorder fashion where a structure is visited and then those contents are passed here in that List. If a pruned field exists then we add that field (and not the original field since that would not be pruned).
OK I think I'm convinced that this change is correct but I'd like @rdblue take on this.
| if (field != null) { | ||
| filteredFields.add(field); | ||
| hasChange = true; | ||
| } else if (fieldId != null && selectedIds.contains(fieldId)) { |
There was a problem hiding this comment.
I don't think that the order should change. If the field itself is selected by ID then the entire field should be projected. For instance, you could have a case where multiple fields are referenced:
SELECT response FROM http_log WHERE response.status = 200That references response.status specifically, but the entire response struct should be produced.
I think I do agree with the secondary case where if the field was returned from traversing the schema, then at least one sub-field was projected and the projected version should be returned.
There was a problem hiding this comment.
When we select the entire struct column like:
SELECT response FROM http_log
response is expanded into all of its nested fields, and adds every nested field’s id to the selected id list.
By the time we reach this code path, field already matches originalField.
So in the above case you mentioned, it will return correct projection.
There was a problem hiding this comment.
Right @sriharshaj,
so @rdblue given
SELECT response FROM http_log WHERE response.status = 200
selectedIds ends up including the ID from the top level struct and all of it's nested fields. As a result, I think we do first want to check anything from the pruned set of fields. If that's defined AND the whole struct is projected that ends up working anyways because the set of fields for the struct already includes everything from the preorder traversal (i.e. the pruning didn't prune anything since there's nothing to prune from response).
This relies on the fact that selectedIds includes the fields from the top level struct and it's nested fields. I think this is fine because the only interface to PruneColumns is ParquetSchemaUtils.pruneColumns which always includes all these fields anyways in the implementation.
I also wrote a quick test (albeit via Spark) to prove it but this logic is all embedded in our parquet APIs anyways so it applies to any integration using those.
@TestTemplate
public void testStructProjection() {
assumeThat(fileFormat).isEqualTo(FileFormat.PARQUET);
sql("CREATE TABLE %s (response struct<code:int, body:string>) using iceberg", tableName);
sql("insert into %s values named_struct(\"code\", 200, \"body\", \"success\")", tableName);
sql("insert into %s values named_struct(\"code\", 503, \"body\", \"because istio\")", tableName);
List<Object[]> results = sql("SELECT response from %s where response.code > 500", tableName);
assertThat(results).hasSize(1);
assertThat(results.get(0)[0]).isEqualTo(new Object[]{503, "because istio"});
}
There was a problem hiding this comment.
Was re-reading my post, I meant post-order not pre-order (since the child fields are visited and pruned first) but my point still stands I think.
There was a problem hiding this comment.
I think you're right that the current way that this class is called by ParquetSchemaUtil.pruneColumns will always pass the struct IDs in. Here's that method:
public static MessageType pruneColumns(MessageType fileSchema, Schema expectedSchema) {
// column order must match the incoming type, so it doesn't matter that the ids are unordered
Set<Integer> selectedIds = TypeUtil.getProjectedIds(expectedSchema);
return (MessageType)
TypeWithSchemaVisitor.visit(
expectedSchema.asStruct(), fileSchema, new PruneColumns(selectedIds));
}The call to TypeUtil.getProjectedIds calls getIdsInternal(schema.asStruct(), true /*includeStructIds*/). If we keep the previous order of the if blocks as I suggested, then this call would always produce the original struct, which doesn't fix column pruning. However, I think we do want to keep the original order of the blocks and fix the problem by setting includeStructIds to false. That way we don't incorrectly select entire structs unless they are explicitly selected.
I think that the behavior of selecting the entire struct when its field ID is passed in is correct, for the reason I gave in my comment above. I don't think that we should change the behavior of the visitor just because we know how it is called. That creates a dependency for correct behavior on how it is called, which we don't want to assume. We could add other uses of this visitor in the future and it should still behave predictably if we reuse it and don't make the same choices in how it is called.
There was a problem hiding this comment.
@rdblue
From my reading of the code, setting includeStructIds to false doesn’t consistently add the struct field ID to selectedIds. As a result, (fieldId != null && selectedIds.contains(fieldId)) may never evaluate to true.
Also, given the current implementation, when we only look at expectedSchema in:
TypeUtil.getProjectedIds(expectedSchema);
it seems we can’t determine whether a subset of struct fields is selected or the entire struct is selected, since there’s no table schema to diff against.
In addition, even if only one nested field of a struct is selected, it looks like org.apache.iceberg.Schema can’t be built without the parent struct field ID.
One possible way to distinguish "entire struct selected" from "subset selected" would be to omit the struct’s nested fields from the Schema when the full struct is selected (In this case above change should work).
Please let me know if I am missing anything.
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
@amogh-jahagirdar and @rdblue Can you take a look at PR again? Thanks. |
|
@sriharshaj Hi, |
Hey @IgorBerman, |
|
I was checking this mechanism lately and I think there are 2 parts in the story. |
Yeah, that sounds right. Optimizations on the Spark side may not be applied when reading the data from Parquet. |
ef071bc to
2865afc
Compare
|
ok, I was a bit blind to what you are doing and did something similar(after you opened your PR and actually I commented on it before) due to our usecase (where we want pruning of columns inside structs inside arrays #14744). I needed to support cases of pruning not only due to structs chains but also for struct<array<struct<array>> cases |
|
ok, verified and your fix passes also my tests containing nested arrays of structs |
Thanks for testing. Sure, I will add more of them. |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
Address review feedback: document in PruneColumns and ParquetSchemaUtil.pruneColumns that a partially projected struct is intentional and kept to its projected subtree, resolving the concern raised in apache#12634, and rename the map-value test whose name overstated its nesting depth. Generated-by: Claude Code (Claude Opus 5.5)
This PR addresses a performance issue related to column pruning in nested structures when reading Iceberg tables from Parquet files.
Issue
In the current implementation of
PruneColumns#struct, we are including the entire parent column in the read schema, even when only a subset of nested fields is required. This occurs because:The
originalFieldis being added to thefilteredFieldsregardless of whether the passed filteredfieldis already present. This behavior causes the entire parent column to be read if any leaf node within the nested structure is selected in a query.Proposed solution
I updated the code to use passed filtered
fieldwhen present, and useoriginalFieldwhen passed filteredfieldis null.