Skip to content

perf: read only the nested fields Spark keeps in the native Iceberg scan - #6725

Merged
andygrove merged 9 commits into
apache:mainfrom
comphead:iceberg-nested-schema-pruning
Oct 16, 2026
Merged

andygrove merged 9 commits into
apache:mainfrom
comphead:iceberg-nested-schema-pruning

Conversation

@comphead

@comphead comphead commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #6724.

Rationale for this change

The native Iceberg scan reads and decodes every field of a projected struct, list, or map column, even when Spark's nested schema pruning keeps only some of them. On TPC-H SF1000 with a nested schema, Q20's partsupp scan reads 26.4 GiB where the query needs only ps_availqty. This is the main reason Comet's scan lead shrinks on that schema. #6724 has the measurements.

What changes are included in this PR?

  • CometIcebergNativeScan passes iceberg-rust the pruned scan schema (SparkScan.expectedSchema()) as the task schema, without its metadata columns, instead of the table schema. iceberg-rust then reads only the leaves the query uses. CometIcebergNativeScanMetadata.extract builds that schema during planning, where a reflection failure falls back to Spark.
  • Tasks with deletes use the pruned schema too. Before, they read with task.schema(), which is the current table schema, so a VERSION AS OF read of a column dropped after the snapshot failed natively with field not found when the table had deletes. With the config on, it now matches Spark.
  • Every task of a scan reads with one schema, built during planning in CometIcebergNativeScanMetadata.extract: the pruned scan schema plus the partition sources and equality-delete keys the tasks need beyond the projection, or the scan's full schema plus the equality keys it lacks. iceberg-rust builds an equality-delete predicate once per delete file, with the names of the task that loads it first, and binds it by name in every task of the reader, so every task must name each key the same way.
  • The names come from the scan's full schema: the newest schema, current first, in which every field the scan reads, at any depth, has the same name and type. That is the current schema unless a VERSION AS OF snapshot named or typed a field differently. An equality key dropped before that schema takes the newest name that no other column uses. A scan whose keys cannot be named that way falls back to Spark.
  • Because the full schema carries the snapshot's names too, a VERSION AS OF read of a column renamed after the snapshot matches Spark with the config on or off. Before, the native scan returned NULL for such a column, or the other column's values when two columns swapped names (see Native Iceberg scan returns wrong values for a column renamed after a VERSION AS OF snapshot #6822).
  • The type check keeps the snapshot's types in the full schema, as the pruned scan schema already does. Without it, a VERSION AS OF read of a column promoted after the snapshot (an int to a bigint, a wider decimal, or a nested field) could take a newer schema with the promoted type, which the batch adaptation after iceberg-rust cannot narrow back. Native Iceberg scan fails a VERSION AS OF read of a column whose type was promoted after the snapshot #6993 tracks such reads failing on main and branch-1.1. With this PR, andygrove found that its repros match Spark natively with pruning on and off: an int, float, decimal, list element, map value, list-of-struct field, or struct field promoted after the snapshot, on an unpartitioned table and on one partitioned by a nested field. So do equality deletes keyed on a promoted column. Native Iceberg scan fails a VERSION AS OF read of a column whose type was promoted after the snapshot #6993 stays open for branch-1.1.
  • A scan reads with the full schema when a missing partition source or equality-delete key is nested inside a struct the query pruned. Appending that field at the top level could clash with a top-level column of the same name (Invalid schema: multiple fields for name region). Such a scan gets no pruning. A VERSION AS OF read whose full schema holds a nested field that an older data file lacks falls back to Spark, while main appends the missing partition source or key at the top level and reads it natively. Native Iceberg scan reads its full schema when a partition source is nested in a struct the query prunes #7019 tracks merging the field into its pruned struct instead.
  • Serialization builds the task schema, its pool entry, and the projected field ids once per scan instead of once per task. Planning inspects each distinct partition spec and delete file once.
  • A new config, spark.comet.scan.icebergNative.nestedSchemaPruning.enabled (default true), turns the pruning off.
  • IcebergReflection gains nativeTaskSchemas, withoutMetadataColumns, and MaxDataFieldId, which CometIcebergNativeWrite now shares. findFieldObject lets reflection failures propagate, so planning falls back instead of reading a failure as a missing field.
  • The nested-evolution fallback from fix: fall back from the native Iceberg scan when a nested field was added or renamed #6543 also walks the schema the scan reads (CometIcebergNativeScanMetadata.taskSchema, which serialization uses too). For VERSION AS OF that can be older than the table's schema and hold a nested field an earlier data file lacks. taskSchema is fixed when the metadata is built during planning, so turning pruning off before a planned query runs, as with a cached DataFrame, cannot make the scan read a schema the check did not walk. Its comment in CometScanRule, and one in CometIcebergNativeSuite, now describe the pruned read.
  • The Iceberg user guide lists the new naming fallback under Current limitations.
  • CometIcebergReadBenchmark gets a nested column case. --nested-only runs only that case.

How are these changes tested?

Four new tests in CometIcebergNativeSuite, and new cases in the existing nested-evolution fallback test: a VERSION AS OF read whose full schema still has a nested field the first data file predates falls back, an unpartitioned one that can read the pruned schema stays native unless pruning is off, and that read, planned with pruning on and run with pruning off, still reads the pruned schema:

  • bytes_scanned with the config on and off, on a table whose struct, list, and map columns are mostly wide pad fields.
  • Equality and position deletes on keys the query does not project, a partition source it does not project, and VERSION AS OF. The time travel reads cover columns dropped, renamed, and swapped after a snapshot that has deletes, a partition source renamed to a dropped column's name, and two partition sources renamed into each other's names. Each of these queries also checks that the scan serializes one task schema, without the pruned fields, so tasks with deletes and tasks that append a partition source get the pruned schema, and that the full schema read with pruning off gives the same answer. These checks run on Iceberg 1.8 and later, like the suite's other commonData checks. The test also reads an int, a decimal, and a nested field promoted after a snapshot, where a column the query reads was dropped later, so the full schema must come from the history. It runs those reads on a table partitioned by a nested field and on an unpartitioned one, each with pruning on and off, with one promoted column per query so that another promotion cannot pick the schema for it.
  • Equality deletes in tasks that share a reader, with adaptive split sizing off: a key renamed after another key was dropped, keys from schemas that never held both, and a VERSION AS OF read over two partition specs where two columns swap names after the snapshot. Each runs with pruning on and off. Iceberg 1.11 and later, which equality deletes on dropped columns need.
  • Data files without field ids and no name mapping. Spark's own reader returns NULL nested fields there, so the pruned native read is compared with the full native read.

A new SQL fixture, sql-tests/iceberg/nested_schema_pruning.sql, checks the pruned reads against Spark:

  • struct, nested struct, list, and map pruning, with NULLs at every level, empty lists and maps, and a nested filter
  • a nested partition source next to a top-level column of the same name, which fails without the guard
  • a renamed partition source with a new column under its old name, which fails when the source resolves from the oldest schema

Locally at 170e5eb48d, on the default Spark 4.1 profile with a release native build, CometIcebergNativeSuite passes (120 passed, 1 version-gated cancel), and so do the three Iceberg SQL fixtures. Against the code before 6213d0576e, the new equality-delete test fails with Field x0 not found in schema. The new fallback cases fail without the nested-evolution changes: the partitioned one in the native scan with Incorrect number of arrays for StructArray fields, and the unpartitioned one by falling back when it should stay native. Against the full-schema choice at 5395e50ba3, each promotion read that uses the full schema fails with Parquet column cannot be converted, for example Column: [[x]], Expected: int, Found: INT64. A check on type ids alone misses the decimal, and one on top-level columns alone misses the nested field. With taskSchema read again when the scan runs, the read planned with pruning on and run with pruning off fails with Incorrect number of arrays for StructArray fields, expected 2 got 1.

In CI at 86eaa60d37, the head before the planning-time task schema, every check passed, including the Comet suites on Spark 4.1, the SQL fixture, and the run-iceberg-tests jobs (Iceberg 1.8, 1.9, 1.10, and 1.11). CI also compiles every Spark profile, Scala 2.12 included.

Not run: the S3 suites, Spark's SQL tests, and the Comet suites on the other Spark profiles.

CometIcebergReadBenchmark --nested-only reads 4M rows on one local core with a warm page cache:

Query Spark Comet, every nested field Comet, pruned
explode(partsupp_data.ps_availqty) (Q20 shape) 413 ms 624 ms 62 ms
explode(partsupp_data.ps_comment) (control) 2,800 ms 590 ms 542 ms

Bytes scanned for the first query drop from 124.0 MiB to 7.5 MiB.

Cluster results. TPC-H SF1000 Q20 on a nested schema on Iceberg (Spark 4.0, 32 executors x 16 cores), three iterations per run:

Run Mean
Spark 34.4 s
Comet, this PR 25.9 s
Comet, baseline 33.9 s
  • The partsupp scan reads 2.48 GiB instead of 26.36 GiB, and its stage takes 7.1-8.5 s instead of 18.3-25.2 s in every iteration. It is no longer on Q20's critical path.
  • Iteration 1 with this PR lost about 74 s to one part scan task. That task ran 80.4 s against a 6.5 s median, with the same input as its peers, and nearly all of it was spent outside compute, which looks like a stalled read. Q20 reads part_data whole, so this scan is the same with and without the change.
  • The baseline ran an earlier Comet snapshot, not this PR's base commit. Only the partsupp difference can be credited to this change with confidence.
  • Q20's critical path is now the single-task supplier scan (19.5-20.3 s) followed by the final join stage (4.4-4.8 s).

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Iceberg area:scan Parquet scan / data reading area:writer Native Parquet writer enhancement New feature or request performance run-iceberg-tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native Iceberg scan reads every field of a nested column even when Spark prunes it

4 participants