Repository navigation
Conversation
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS. The new legacy-version distribution support has real CTAS/file-count value, and the merged option/catalog-backed DataSource behavior is covered by current-head validation. Two independently reproduced legacy Spark regressions remain below. All reproduction runs use this exact current head, rather than the earlier Draft revision.
Spark 3.5.9/JDK 8: 82 option/distribution/DataFrame/merge-schema tests passed with normal Maven checks, including merged table/session/DataSource precedence and catalog-backed append/full overwrite. Spark 4.2/JDK 17: all 18 compatibility cases passed, including three additional current-head tests covering the new option precedence, catalog binding, AQE on/off, whole-table overwrite and operation overrides. Spark 3.4.4/JDK 8: the driver-interrupted CTAS V1 cleanup control passes, while the V2 cleanup assertion fails. Spark 3.2.4/JDK 8: both real write/read/file-count controls pass, but V2 fails the AQE task-count bound (200 versus V1's 1). All runs retained normal Maven checks. Diagnostic sources are isolated; no production edits were made. Current CI is queued, so no full green CI is claimed.
| } | ||
| Nil | ||
| } catch { | ||
| case NonFatal(error) => |
There was a problem hiding this comment.
[P2] Run CTAS failure cleanup when the driver is interrupted
Scala NonFatal excludes InterruptedException, so this skips both dropTable and abortStagedChanges when the driver thread is interrupted after table creation. On this exact head with Spark 3.4.4/JDK 8, a real CTAS driver-interruption probe propagates the original InterruptedException for both routes: the V1 control removes the created table, while V2 leaves it registered. The assertion checks the exception before any cancelJobGroup cleanup, so this is not a task failure wrapped in SparkException. A retry then encounters the leftover table. Preserve Spark's native abort/drop behavior on interruption as well as ordinary exceptions, while propagating the original interruption. This helper is used by the new Spark 3.2–3.4 table-as-select path; runtime verification here is on 3.4.
| RepartitionByExpression( | ||
| clustered.clustering().map(bind).toSeq, | ||
| query, | ||
| conf.numShufflePartitions) |
There was a problem hiding this comment.
[P2] Keep adaptive coalescing available for legacy V2 writes
Passing an explicit partition count creates a REPARTITION_BY_NUM exchange on Spark 3.2/3.3; their CoalesceShufflePartitions rule does not coalesce that origin. The previous V1 repartition(columns) path remains adaptive. On this exact head with Spark 3.2.4/JDK 8, a real one-bucket write of 40 rows from four input partitions, with AQE/coalescing enabled and shuffle.partitions=200, submits stages [4,1] through V1 versus [4,200] through V2. Both read back the same 40 rows and one actual data file. The newly default-enabled V2 route therefore schedules 199 unnecessary write tasks for this small single-bucket input. Construct an unspecified-count repartition through the version-specific adapter rather than freezing conf.numShufflePartitions, and cover actual AQE stage counts.
Purpose
Enable
write.use-v2-writeby default for supported batch writes, removing the version-dependent default and blanket pre-3.4 disablement. Resolve the switch from merged options so persistent table properties, global/table-specific session settings, and DataSource write options follow the existing option precedence. Unsupported table configurations, location-based writes without a catalog, and streaming writes continue to use V1.For DataSource writes with a
catalogoption, attach the loading catalog and identifier to the write target relation. This lets Spark resolve Paimon's bucket function during V2 distribution planning. Cover append and full-table overwrite, including when dynamic partition overwrite is configured in the session.Spark 3.2/3.3 do not pass a function catalog when resolving V2 write distributions. Bind the equivalent distribution with Paimon's bucket expression before native write planning. Keep the partition count unspecified through the version-specific compatibility layer so AQE can coalesce the shuffle. A one-bucket write with four input partitions and 200 shuffle partitions runs four input tasks and one write task for both V1 and V2.
Spark 3.2-3.4 table-as-select executors also bypass normal distribution preparation, so route their writes through append planning after table creation/staging, covering both
SparkCatalogandSparkGenericCatalog. Preserve native creation/replacement logic and commit/abort callbacks; use named execution nodes whose constructors support Spark's reflective plan cloning. Use Spark's native failure callback helper so driver interruption also aborts staged tables or drops newly created tables while preserving the original exception and suppressed cleanup errors.This keeps records for the same partition/bucket together. For example, a CTAS with one bucket and four input partitions produces one data file instead of four.
Tests
InterruptedExceptionand remove the table before executor job cancellation.spark4Maven profile, coveringPaimonOptionTestandV2WriteRequireDistributionTest. Four legacy-only interruption cases were skipped by their version guards.spark3Maven profile. Formatting, Checkstyle, and Maven validation ran without thefast-buildprofile;git diff --checkpassed. All 316 tests above were rerun with the final changes.