Skip to content

[AURON #2634] Support building auron-paimon against Paimon 2.0 - #2637

Open
COLDMANYU wants to merge 6 commits into
apache:masterfrom
COLDMANYU:paimon-2.0-build
Open

COLDMANYU wants to merge 6 commits into
apache:masterfrom
COLDMANYU:paimon-2.0-build

Conversation

@COLDMANYU

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #2634

Rationale for this change

auron-paimon only builds against Paimon 1.2. Paimon 1.3+ is binary incompatible with code
compiled against 1.2: DataFileMeta became an interface in 1.3 (apache/paimon#6134), and the
paimon-spark artifacts gained a Scala suffix. On a Paimon 2.0 runtime the current jar fails at
planning time with IncompatibleClassChangeError. That is a LinkageError, so the conversion
fallback does not catch it and the query fails.

What changes are included in this PR?

Build:

  • New paimon-2.0 profile (Paimon 2.0.0, paimon-spark-${shortSparkVersion}_${scalaVersion}).
    paimon-1.2 is unchanged and stays the default in the build examples.
  • Version-specific sources: src/{main,test}/paimon-<version>/scala, selected by the new
    paimonSourceDir property and added with build-helper-maven-plugin. They are included in
    spotless, scalastyle and dev/reformat. This PR adds no version-specific source yet. It only
    prepares the layout for features that need Paimon >= 1.3 APIs (see [AIP] Native write for Paimon append-only tables #2636).
  • auron-build.sh --paimon 2.0, the Paimon CI matrix (["1.2", "2.0"]) and the build matrix in
    CONTRIBUTING.md. Spark 4.0/4.1 rows keep 1.2; see "Not covered" below.

Behaviour:

  • __paimon_file_path of the native V2 scan now follows the Paimon version on the classpath.
    [core] Do not use path.toUri().toString() when a location string is expected internally paimon#7098 (1.4) changed the value from path.toUri.toString to path.toString; they
    differ for paths with characters that need escaping, e.g. % in a partition value. Without
    this, the native scan returns a different value than Paimon's own reader on Paimon >= 1.4.
    The version comes from the Implementation-Version of the Paimon jar's manifest; if it cannot
    be read, the newer behaviour is used.

Are there any user-facing changes?

  • New build option --paimon 2.0 / -Ppaimon,paimon-2.0. The artifact must match the Paimon
    version that is deployed at runtime. This is documented in CONTRIBUTING.md.
  • No change for paimon-1.2 builds. The __paimon_file_path value changes only on Paimon >= 1.4,
    where it now matches Paimon.

How was this patch tested?

Spark 3.5.8, Scala 2.12, JDK 8:

  • AuronPaimonV2IntegrationSuite (21 tests) passes unchanged under both -Ppaimon-1.2 and
    -Ppaimon-2.0. The suite is shared, so both profiles run the same read coverage.
  • Spotless/scalastyle pass for both profiles.
  • Native scan on a production Paimon 2.0 table (2,630,709,474 rows, 197 columns): row count and a
    hash over all columns are identical to the Paimon reader.

Not covered:

  • Spark 4.x with Paimon 2.0: paimon-spark-4.0_2.13 / paimon-spark-4.1_2.13 2.0.0 exist on
    Maven Central, but I have only verified Spark 3.5, so the matrix only lists 2.0 for Spark 3.5.
    Paimon 1.2 never published Spark 4 artifacts, so I left the existing 1.2 entries for
    Spark 4.x as they are.

Was this patch authored or co-authored using generative AI tooling?

  • Yes
  • No

Generated-by: Hermes Agent 0.22.2 (Claude Opus 5.5)

… 2.0

Paimon 1.3+ is binary incompatible with code compiled against 1.2
(DataFileMeta became an interface and DataFileMeta.forAppend gained
parameters), so a module compiled against 1.2 fails at planning time with
IncompatibleClassChangeError/NoSuchMethodError on Paimon 2.0. Such
LinkageErrors are not caught by the conversion fallback.

- add a paimon-2.0 profile compiling the module against Paimon 2.0.0, using
  the Scala-suffixed paimon-spark artifact introduced in 1.3
- support Paimon version specific sources (src/{main,test}/paimon-<ver>/scala)
  and include them in spotless/scalastyle and dev/reformat
- expose the profile through auron-build.sh and the Paimon CI matrix
…th Paimon 1.4+

apache/paimon#7098 (Paimon 1.4) changed the __paimon_file_path metadata
column from path.toUri.toString to path.toString. They differ for paths
with characters that need escaping (e.g. a '%' in a partition value), so
the native scan returned a different value than the Paimon reader on
Paimon 2.0. Follow the Paimon version found on the classpath.
val majorMinor = version.split("[.-]").take(2).flatMap(s => scala.util.Try(s.toInt).toOption)
majorMinor match {
case Array(major, minor) => major < 1 || (major == 1 && minor < 4)
case _ => false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When Implementation-Version is missing, this falls back to Path.toString, even on Paimon 1.2. I reproduced the mismatch with a Paimon 1.2 JAR with its manifest removed.

For a partition path containing %, the Paimon reader returned p=50%2525, while the native scan’s new formatting logic returned p=50%25 for __paimon_file_path.

Could we use the version-specific source directories introduced in this PR to select the path format at build time?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks for reproducing it. Fixed in 6c93c66: the format is now chosen at build time with the version specific source directories. PaimonFilePathFormat in src/main/paimon-1.2 uses path.toUri.toString, and the one in src/main/paimon-2.0 uses path.toString. The manifest lookup is gone, so a missing Implementation-Version no longer matters. The existing 50% partition test passes with both -Ppaimon-1.2 and -Ppaimon-2.0.

fail-fast: false
matrix:
paimon: [ "1.2" ]
paimon: [ "1.2", "2.0" ]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If both Paimon matrix jobs fail and generate test reports, they’ll try to upload them under the same artifact name. Could we include the Paimon version in the name, for example auron-paimon-test-report-${{ matrix.paimon }} ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 6c93c66, the artifact is now auron-paimon-test-report-${{ matrix.paimon }}.

@slfan1989 slfan1989 self-assigned this Nov 14, 2026
Detecting the Paimon version from the jar manifest at runtime fell back to the
Paimon 1.4+ format when Implementation-Version is missing, which is wrong on
Paimon 1.2. Use the Paimon version specific source directories instead, and
give the test report artifact of each Paimon CI job its own name.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support building auron-paimon against Paimon 2.0

2 participants