Repository navigation
Conversation
… 2.0
Paimon 1.3+ is binary incompatible with code compiled against 1.2
(DataFileMeta became an interface and DataFileMeta.forAppend gained
parameters), so a module compiled against 1.2 fails at planning time with
IncompatibleClassChangeError/NoSuchMethodError on Paimon 2.0. Such
LinkageErrors are not caught by the conversion fallback.
- add a paimon-2.0 profile compiling the module against Paimon 2.0.0, using
the Scala-suffixed paimon-spark artifact introduced in 1.3
- support Paimon version specific sources (src/{main,test}/paimon-<ver>/scala)
and include them in spotless/scalastyle and dev/reformat
- expose the profile through auron-build.sh and the Paimon CI matrix
…th Paimon 1.4+ apache/paimon#7098 (Paimon 1.4) changed the __paimon_file_path metadata column from path.toUri.toString to path.toString. They differ for paths with characters that need escaping (e.g. a '%' in a partition value), so the native scan returned a different value than the Paimon reader on Paimon 2.0. Follow the Paimon version found on the classpath.
| val majorMinor = version.split("[.-]").take(2).flatMap(s => scala.util.Try(s.toInt).toOption) | ||
| majorMinor match { | ||
| case Array(major, minor) => major < 1 || (major == 1 && minor < 4) | ||
| case _ => false |
There was a problem hiding this comment.
When Implementation-Version is missing, this falls back to Path.toString, even on Paimon 1.2. I reproduced the mismatch with a Paimon 1.2 JAR with its manifest removed.
For a partition path containing %, the Paimon reader returned p=50%2525, while the native scan’s new formatting logic returned p=50%25 for __paimon_file_path.
Could we use the version-specific source directories introduced in this PR to select the path format at build time?
There was a problem hiding this comment.
Good catch, thanks for reproducing it. Fixed in 6c93c66: the format is now chosen at build time with the version specific source directories. PaimonFilePathFormat in src/main/paimon-1.2 uses path.toUri.toString, and the one in src/main/paimon-2.0 uses path.toString. The manifest lookup is gone, so a missing Implementation-Version no longer matters. The existing 50% partition test passes with both -Ppaimon-1.2 and -Ppaimon-2.0.
| fail-fast: false | ||
| matrix: | ||
| paimon: [ "1.2" ] | ||
| paimon: [ "1.2", "2.0" ] |
There was a problem hiding this comment.
If both Paimon matrix jobs fail and generate test reports, they’ll try to upload them under the same artifact name. Could we include the Paimon version in the name, for example auron-paimon-test-report-${{ matrix.paimon }} ?
There was a problem hiding this comment.
Done in 6c93c66, the artifact is now auron-paimon-test-report-${{ matrix.paimon }}.
Detecting the Paimon version from the jar manifest at runtime fell back to the Paimon 1.4+ format when Implementation-Version is missing, which is wrong on Paimon 1.2. Use the Paimon version specific source directories instead, and give the test report artifact of each Paimon CI job its own name.
Which issue does this PR close?
Closes #2634
Rationale for this change
auron-paimon only builds against Paimon 1.2. Paimon 1.3+ is binary incompatible with code
compiled against 1.2:
DataFileMetabecame an interface in 1.3 (apache/paimon#6134), and thepaimon-sparkartifacts gained a Scala suffix. On a Paimon 2.0 runtime the current jar fails atplanning time with
IncompatibleClassChangeError. That is aLinkageError, so the conversionfallback does not catch it and the query fails.
What changes are included in this PR?
Build:
paimon-2.0profile (Paimon 2.0.0,paimon-spark-${shortSparkVersion}_${scalaVersion}).paimon-1.2is unchanged and stays the default in the build examples.src/{main,test}/paimon-<version>/scala, selected by the newpaimonSourceDirproperty and added withbuild-helper-maven-plugin. They are included inspotless, scalastyle and
dev/reformat. This PR adds no version-specific source yet. It onlyprepares the layout for features that need Paimon >= 1.3 APIs (see [AIP] Native write for Paimon append-only tables #2636).
auron-build.sh --paimon 2.0, the Paimon CI matrix (["1.2", "2.0"]) and the build matrix inCONTRIBUTING.md. Spark 4.0/4.1 rows keep
1.2; see "Not covered" below.Behaviour:
__paimon_file_pathof the native V2 scan now follows the Paimon version on the classpath.[core] Do not use
path.toUri().toString()when a location string is expected internally paimon#7098 (1.4) changed the value frompath.toUri.toStringtopath.toString; theydiffer for paths with characters that need escaping, e.g.
%in a partition value. Withoutthis, the native scan returns a different value than Paimon's own reader on Paimon >= 1.4.
The version comes from the
Implementation-Versionof the Paimon jar's manifest; if it cannotbe read, the newer behaviour is used.
Are there any user-facing changes?
--paimon 2.0/-Ppaimon,paimon-2.0. The artifact must match the Paimonversion that is deployed at runtime. This is documented in CONTRIBUTING.md.
paimon-1.2builds. The__paimon_file_pathvalue changes only on Paimon >= 1.4,where it now matches Paimon.
How was this patch tested?
Spark 3.5.8, Scala 2.12, JDK 8:
AuronPaimonV2IntegrationSuite(21 tests) passes unchanged under both-Ppaimon-1.2and-Ppaimon-2.0. The suite is shared, so both profiles run the same read coverage.hash over all columns are identical to the Paimon reader.
Not covered:
paimon-spark-4.0_2.13/paimon-spark-4.1_2.132.0.0 exist onMaven Central, but I have only verified Spark 3.5, so the matrix only lists 2.0 for Spark 3.5.
Paimon 1.2 never published Spark 4 artifacts, so I left the existing
1.2entries forSpark 4.x as they are.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Hermes Agent 0.22.2 (Claude Opus 5.5)