Skip to content

[spark] Enable V2 batch writes by default - #11190

Open
Zouxxyy wants to merge 4 commits into
apache:masterfrom
Zouxxyy:feat/spark-v2-write-default
Open

Zouxxyy wants to merge 4 commits into
apache:masterfrom
Zouxxyy:feat/spark-v2-write-default

Conversation

@Zouxxyy

@Zouxxyy Zouxxyy commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Enable write.use-v2-write by default for supported batch writes, removing the version-dependent default and blanket pre-3.4 disablement. Resolve the switch from merged options so persistent table properties, global/table-specific session settings, and DataSource write options follow the existing option precedence. Unsupported table configurations, location-based writes without a catalog, and streaming writes continue to use V1.

For DataSource writes with a catalog option, attach the loading catalog and identifier to the write target relation. This lets Spark resolve Paimon's bucket function during V2 distribution planning. Cover append and full-table overwrite, including when dynamic partition overwrite is configured in the session.

Spark 3.2/3.3 do not pass a function catalog when resolving V2 write distributions. Bind the equivalent distribution with Paimon's bucket expression before native write planning. Keep the partition count unspecified through the version-specific compatibility layer so AQE can coalesce the shuffle. A one-bucket write with four input partitions and 200 shuffle partitions runs four input tasks and one write task for both V1 and V2.

Spark 3.2-3.4 table-as-select executors also bypass normal distribution preparation, so route their writes through append planning after table creation/staging, covering both SparkCatalog and SparkGenericCatalog. Preserve native creation/replacement logic and commit/abort callbacks; use named execution nodes whose constructors support Spark's reflective plan cloning. Use Spark's native failure callback helper so driver interruption also aborts staged tables or drops newly created tables while preserving the original exception and suppressed cleanup errors.

This keeps records for the same partition/bucket together. For example, a CTAS with one bucket and four input partitions produces one data file instead of four.

Tests

  • 316 tests passed on Spark 3.2.4, 3.3.4, 3.4.4, and 3.5.9, covering option defaults and precedence, capability and path fallbacks, catalog-backed DataSource writes, regular writes, CTAS/RTAS, AQE on/off, bucket file counts, failed CTAS cleanup, and row-level operations. Eighteen cases were skipped by their version guards.
  • New regression tests compare actual V1/V2 stage task counts with AQE enabled and interrupt real CTAS driver threads for both ordinary and staged tables. On Spark 3.2-3.4, all interruption cases preserve the original InterruptedException and remove the table before executor job cancellation.
  • 32 compatibility tests passed on Spark 4.1.4 with Java 17, Scala 2.13, and the spark4 Maven profile, covering PaimonOptionTest and V2WriteRequireDistributionTest. Four legacy-only interruption cases were skipped by their version guards.
  • Spark 3 tests used Java 8 and the spark3 Maven profile. Formatting, Checkstyle, and Maven validation ran without the fast-build profile; git diff --check passed. All 316 tests above were rerun with the final changes.

@Zouxxyy
Zouxxyy marked this pull request as draft November 16, 2026 06:32
@Zouxxyy
Zouxxyy marked this pull request as ready for review November 16, 2026 07:19

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirement fit: SUPPORTED. Implementation: FINDINGS. The new legacy-version distribution support has real CTAS/file-count value, and the merged option/catalog-backed DataSource behavior is covered by current-head validation. Two independently reproduced legacy Spark regressions remain below. All reproduction runs use this exact current head, rather than the earlier Draft revision.

Spark 3.5.9/JDK 8: 82 option/distribution/DataFrame/merge-schema tests passed with normal Maven checks, including merged table/session/DataSource precedence and catalog-backed append/full overwrite. Spark 4.2/JDK 17: all 18 compatibility cases passed, including three additional current-head tests covering the new option precedence, catalog binding, AQE on/off, whole-table overwrite and operation overrides. Spark 3.4.4/JDK 8: the driver-interrupted CTAS V1 cleanup control passes, while the V2 cleanup assertion fails. Spark 3.2.4/JDK 8: both real write/read/file-count controls pass, but V2 fails the AQE task-count bound (200 versus V1's 1). All runs retained normal Maven checks. Diagnostic sources are isolated; no production edits were made. Current CI is queued, so no full green CI is claimed.

}
Nil
} catch {
case NonFatal(error) =>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Run CTAS failure cleanup when the driver is interrupted

Scala NonFatal excludes InterruptedException, so this skips both dropTable and abortStagedChanges when the driver thread is interrupted after table creation. On this exact head with Spark 3.4.4/JDK 8, a real CTAS driver-interruption probe propagates the original InterruptedException for both routes: the V1 control removes the created table, while V2 leaves it registered. The assertion checks the exception before any cancelJobGroup cleanup, so this is not a task failure wrapped in SparkException. A retry then encounters the leftover table. Preserve Spark's native abort/drop behavior on interruption as well as ordinary exceptions, while propagating the original interruption. This helper is used by the new Spark 3.2–3.4 table-as-select path; runtime verification here is on 3.4.

Comment on lines +71 to +74
RepartitionByExpression(
clustered.clustering().map(bind).toSeq,
query,
conf.numShufflePartitions)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep adaptive coalescing available for legacy V2 writes

Passing an explicit partition count creates a REPARTITION_BY_NUM exchange on Spark 3.2/3.3; their CoalesceShufflePartitions rule does not coalesce that origin. The previous V1 repartition(columns) path remains adaptive. On this exact head with Spark 3.2.4/JDK 8, a real one-bucket write of 40 rows from four input partitions, with AQE/coalescing enabled and shuffle.partitions=200, submits stages [4,1] through V1 versus [4,200] through V2. Both read back the same 40 rows and one actual data file. The newly default-enabled V2 route therefore schedules 199 unnecessary write tasks for this small single-bucket input. Construct an unspecified-count repartition through the version-specific adapter rather than freezing conf.numShufflePartitions, and cover actual AQE stage counts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants