Repository navigation
perf: read only the nested fields Spark keeps in the native Iceberg scan - #6725
Merged
Merged
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #6724.
Rationale for this change
The native Iceberg scan reads and decodes every field of a projected struct, list, or map column, even when Spark's nested schema pruning keeps only some of them. On TPC-H SF1000 with a nested schema, Q20's
partsuppscan reads 26.4 GiB where the query needs onlyps_availqty. This is the main reason Comet's scan lead shrinks on that schema. #6724 has the measurements.What changes are included in this PR?
CometIcebergNativeScanpasses iceberg-rust the pruned scan schema (SparkScan.expectedSchema()) as the task schema, without its metadata columns, instead of the table schema. iceberg-rust then reads only the leaves the query uses.CometIcebergNativeScanMetadata.extractbuilds that schema during planning, where a reflection failure falls back to Spark.task.schema(), which is the current table schema, so aVERSION AS OFread of a column dropped after the snapshot failed natively withfield not foundwhen the table had deletes. With the config on, it now matches Spark.CometIcebergNativeScanMetadata.extract: the pruned scan schema plus the partition sources and equality-delete keys the tasks need beyond the projection, or the scan's full schema plus the equality keys it lacks. iceberg-rust builds an equality-delete predicate once per delete file, with the names of the task that loads it first, and binds it by name in every task of the reader, so every task must name each key the same way.VERSION AS OFsnapshot named or typed a field differently. An equality key dropped before that schema takes the newest name that no other column uses. A scan whose keys cannot be named that way falls back to Spark.VERSION AS OFread of a column renamed after the snapshot matches Spark with the config on or off. Before, the native scan returned NULL for such a column, or the other column's values when two columns swapped names (see Native Iceberg scan returns wrong values for a column renamed after a VERSION AS OF snapshot #6822).VERSION AS OFread of a column promoted after the snapshot (an int to a bigint, a wider decimal, or a nested field) could take a newer schema with the promoted type, which the batch adaptation after iceberg-rust cannot narrow back. Native Iceberg scan fails a VERSION AS OF read of a column whose type was promoted after the snapshot #6993 tracks such reads failing onmainandbranch-1.1. With this PR, andygrove found that its repros match Spark natively with pruning on and off: an int, float, decimal, list element, map value, list-of-struct field, or struct field promoted after the snapshot, on an unpartitioned table and on one partitioned by a nested field. So do equality deletes keyed on a promoted column. Native Iceberg scan fails a VERSION AS OF read of a column whose type was promoted after the snapshot #6993 stays open forbranch-1.1.Invalid schema: multiple fields for name region). Such a scan gets no pruning. AVERSION AS OFread whose full schema holds a nested field that an older data file lacks falls back to Spark, whilemainappends the missing partition source or key at the top level and reads it natively. Native Iceberg scan reads its full schema when a partition source is nested in a struct the query prunes #7019 tracks merging the field into its pruned struct instead.spark.comet.scan.icebergNative.nestedSchemaPruning.enabled(defaulttrue), turns the pruning off.IcebergReflectiongainsnativeTaskSchemas,withoutMetadataColumns, andMaxDataFieldId, whichCometIcebergNativeWritenow shares.findFieldObjectlets reflection failures propagate, so planning falls back instead of reading a failure as a missing field.CometIcebergNativeScanMetadata.taskSchema, which serialization uses too). ForVERSION AS OFthat can be older than the table's schema and hold a nested field an earlier data file lacks.taskSchemais fixed when the metadata is built during planning, so turning pruning off before a planned query runs, as with a cachedDataFrame, cannot make the scan read a schema the check did not walk. Its comment inCometScanRule, and one inCometIcebergNativeSuite, now describe the pruned read.CometIcebergReadBenchmarkgets a nested column case.--nested-onlyruns only that case.How are these changes tested?
Four new tests in
CometIcebergNativeSuite, and new cases in the existing nested-evolution fallback test: aVERSION AS OFread whose full schema still has a nested field the first data file predates falls back, an unpartitioned one that can read the pruned schema stays native unless pruning is off, and that read, planned with pruning on and run with pruning off, still reads the pruned schema:bytes_scannedwith the config on and off, on a table whose struct, list, and map columns are mostly widepadfields.VERSION AS OF. The time travel reads cover columns dropped, renamed, and swapped after a snapshot that has deletes, a partition source renamed to a dropped column's name, and two partition sources renamed into each other's names. Each of these queries also checks that the scan serializes one task schema, without the pruned fields, so tasks with deletes and tasks that append a partition source get the pruned schema, and that the full schema read with pruning off gives the same answer. These checks run on Iceberg 1.8 and later, like the suite's othercommonDatachecks. The test also reads an int, a decimal, and a nested field promoted after a snapshot, where a column the query reads was dropped later, so the full schema must come from the history. It runs those reads on a table partitioned by a nested field and on an unpartitioned one, each with pruning on and off, with one promoted column per query so that another promotion cannot pick the schema for it.VERSION AS OFread over two partition specs where two columns swap names after the snapshot. Each runs with pruning on and off. Iceberg 1.11 and later, which equality deletes on dropped columns need.A new SQL fixture,
sql-tests/iceberg/nested_schema_pruning.sql, checks the pruned reads against Spark:Locally at
170e5eb48d, on the default Spark 4.1 profile with a release native build,CometIcebergNativeSuitepasses (120 passed, 1 version-gated cancel), and so do the three Iceberg SQL fixtures. Against the code before6213d0576e, the new equality-delete test fails withField x0 not found in schema. The new fallback cases fail without the nested-evolution changes: the partitioned one in the native scan withIncorrect number of arrays for StructArray fields, and the unpartitioned one by falling back when it should stay native. Against the full-schema choice at5395e50ba3, each promotion read that uses the full schema fails withParquet column cannot be converted, for exampleColumn: [[x]], Expected: int, Found: INT64. A check on type ids alone misses the decimal, and one on top-level columns alone misses the nested field. WithtaskSchemaread again when the scan runs, the read planned with pruning on and run with pruning off fails withIncorrect number of arrays for StructArray fields, expected 2 got 1.In CI at
86eaa60d37, the head before the planning-time task schema, every check passed, including the Comet suites on Spark 4.1, the SQL fixture, and therun-iceberg-testsjobs (Iceberg 1.8, 1.9, 1.10, and 1.11). CI also compiles every Spark profile, Scala 2.12 included.Not run: the S3 suites, Spark's SQL tests, and the Comet suites on the other Spark profiles.
CometIcebergReadBenchmark --nested-onlyreads 4M rows on one local core with a warm page cache:explode(partsupp_data.ps_availqty)(Q20 shape)explode(partsupp_data.ps_comment)(control)Bytes scanned for the first query drop from 124.0 MiB to 7.5 MiB.
Cluster results. TPC-H SF1000 Q20 on a nested schema on Iceberg (Spark 4.0, 32 executors x 16 cores), three iterations per run:
partsuppscan reads 2.48 GiB instead of 26.36 GiB, and its stage takes 7.1-8.5 s instead of 18.3-25.2 s in every iteration. It is no longer on Q20's critical path.partscan task. That task ran 80.4 s against a 6.5 s median, with the same input as its peers, and nearly all of it was spent outside compute, which looks like a stalled read. Q20 readspart_datawhole, so this scan is the same with and without the change.partsuppdifference can be credited to this change with confidence.supplierscan (19.5-20.3 s) followed by the final join stage (4.4-4.8 s).