Skip to content

Spark 4.2: Remove plan-time shredded variant checks from vectorized reads - #19009

Open
nssalian wants to merge 2 commits into
apache:mainfrom
nssalian:spark-4.2-remove-shredded-variant-plan-checks
Open

nssalian wants to merge 2 commits into
apache:mainfrom
nssalian:spark-4.2-remove-shredded-variant-plan-checks

Conversation

@nssalian

@nssalian nssalian commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

#16292 added plan-time checks in SparkBatch because the vectorized reader could not decode shredded VARIANT typed_value: a scan goes to the row reader when the table sets write.parquet.shred-variants=true, the variant column uses none or counts metrics, or a file has variant lower bounds, which SparkScanBuilder loads stats for. These checks guess the file layout and miss both ways: shredded files without the property and without variant bounds, such as arrays, failed (#18601), and unshredded files in such tables never vectorize. #18851 makes the reader handle shredded files per file, so this removes the checks and the stats loading. Those scans follow the same vectorization rules as unshredded variants, and disabling vectorization still selects the row reader.

Part of #18669.

Test plan: Spark 4.2 TestSparkVariantRead asserts the scan is columnar only when vectorization is enabled. These fail without this change: readShreddedEnabledByTableProperty (property), testReadShreddedWithMetricsDisabled (none and counts, with the property also set), readMixedObjectShreddedAndUnshreddedFiles and the vectorized case of testReadShreddedAfterPropertyToggled (variant bounds, batch scan), readShreddedIncrementalAppendScan (incremental append scan) and readShreddedVariantAfterDelete (merge-on-read delete). The new tests compare vectorized and row-reader bytes.


AI Disclosure

  • Model: Claude Opus 5.5
  • Platform/Tool: Claude Code
  • Human Oversight: fully reviewed with tests run locally
  • Prompt Summary: Remove the Spark plan-time shredded variant checks that the per-file vectorized reader makes redundant, with Spark tests asserting columnar scans. The code changes and tests are generated.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant