Skip to content

[deps](paimon) Upgrade paimon-rust to 0.5.0 - #70598

Merged
morningman merged 4 commits into
apache:masterfrom
suxiaogang223:codex/paimon-rust-0.5.0
Nov 17, 2026
Merged

morningman merged 4 commits into
apache:masterfrom
suxiaogang223:codex/paimon-rust-0.5.0

Conversation

@suxiaogang223

@suxiaogang223 suxiaogang223 commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Background

Doris uses paimon-rust for native Paimon reads. Paimon and Lance are linked into the BE as Rust static libraries.

What problem does this PR solve?

Issue Number: None
Related PR: #69886 (Rust write integration)
PR Type: Release/Dependency

Problem Summary:

Upgrade to the official paimon-rust 0.5.0 release. Update both libraries to its required Rust 1.95.0 baseline and use Doris's OpenSSL prefix when building Paimon.

Details

  • Versions: paimon-rust 0.5.0-rc1 → 0.5.0; Rust 1.94.0 → 1.95.0.
  • Upstream changes: Adopt the official release without local patches.
  • Compatibility: Rebuild Paimon and Lance together, or refresh the compilation image.
  • Rollback: Revert and rebuild both libraries with Rust 1.94.0.
  • Behavior change: Native Paimon reads use version 0.5.0.

Testing

Passed locally on Linux x86_64:

  • Paimon/Lance third-party builds and Doris ASAN BE/FE builds, including required Java plugins.
  • All 19 existing Paimon build-script tests.
  • Paimon compatibility and basic write/read cases: append, partitioned writes, empty writes and primary-key overwrite. Writes use JNI; Profile confirms Rust reader execution.

Release note

Upgrade paimon-rust to 0.5.0. Building Paimon and Lance requires Rust 1.95.0 or newer, using the same toolchain for both.

Check List (For Author)

  • Behavior changed: Yes (see Details).
  • Does this need documentation: No.
  • Pre-submission status: Partially verified: local checks passed; CI pending.

AI Disclosure

  • Tool: Codex.
  • Generated: build changes, test adaptation and PR description.
  • Human review: Pending author and maintainer review.

suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Nov 16, 2026
### Background

The third-party workflow has a fast script-test job that runs without compiling Rust or downloading native dependencies. The Paimon build-script tests exercise the real build functions against local stubs.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#70598 (dependency upgrade)
PR Type: Test

Problem Summary:

Run the Paimon build-script suite in the existing CI script-test job so the new toolchain and installation checks continue to be verified after the upgrade.

### Details

- **Scenarios:** Old toolchains, stale installed artifacts, incomplete publication, selected Cargo, and offline metadata generation.
- **What existing tests miss:** The workflow currently exercises other dependency scripts without invoking the Paimon build-script suite.
- **Fails when broken:** The added checks fail against the previous Rust 1.94 build scripts.

### Testing

All 28 Paimon build-script tests passed locally. git diff --check passed.

### Release note

None

### Check List (For Author)

- Behavior changed: No.
- Does this need documentation: No.
- Pre-submission status: Ready; CI is pending.

### AI Disclosure

- Tool: Codex.
- Generated: workflow invocation and this description.
- Human review: Pending author and maintainer review.
@suxiaogang223
suxiaogang223 marked this pull request as ready for review November 16, 2026 04:37
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Nov 16, 2026
### Background

The Paimon upgrade changes the release archive and moves both Rust libraries to the Rust 1.95.0 baseline. The existing build scripts already select and install the required toolchain.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#70598 (simplifies the dependency upgrade)
PR Type: Refactor

Problem Summary:

Keep this upgrade focused by removing the added installation receipts and Rust installation mock scaffolding. Upgrading the prebuilt dependencies remains an explicit paired rebuild of Paimon and Lance.

### Details

- **Moves vs. logic changes:** Remove the new receipt publication and rebuild conditions. Restore the existing test structure, update its toolchain fixture, and check the Doris OpenSSL prefix in the existing Cargo stub.
- **Behavior change:** Missing version receipts no longer force rebuilding all third-party dependencies. Existing toolchain selection and interrupted-install handling remain unchanged.
- **Compatibility:** Rebuild Paimon and Lance together with Rust 1.95.0 or refresh the compilation image.
- **Performance impact:** Avoid an automatic full dependency rebuild based on new receipt files.
- **Rollback:** Revert this commit to restore the receipt mechanism and additional tests.

### Testing

All 19 existing Paimon build-script tests passed. Shell syntax checks and git diff --check passed. The native versions and compiler inputs are unchanged from the previous successful dependency, ASAN BE/FE and Paimon write/read validations; those builds were not repeated for this simplification.

### Release note

None

### Check List (For Author)

- Behavior changed: Yes (see Details).
- Does this need documentation: No; the paired rebuild command is in the PR description.
- Pre-submission status: Partially verified: local script checks passed; CI for this revision is pending.

### AI Disclosure

- Tool: Codex.
- Generated: simplification, existing test adaptation, and this description.
- Human review: Pending author and maintainer review.
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Nov 16, 2026
### Background

The Paimon build-script suite exists on master and can be run locally. This dependency upgrade had also added it to the third-party workflow.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#70598 (reduces upgrade scope)
PR Type: Refactor

Problem Summary:

Keep the dependency upgrade focused by restoring the workflow to master. Adding existing tests to CI can be handled separately.

### Details

- **Moves vs. logic changes:** Remove the new workflow invocation; no build script or test implementation changes.
- **Behavior change:** No dependency build behavior change. The workflow no longer adds the Paimon script suite.
- **Compatibility:** Unchanged.
- **Performance impact:** None for Doris or dependency compilation.
- **Rollback:** Restore the workflow invocation.

### Testing

Verified that the workflow matches master and git diff --check passes. The 19 script tests previously passed locally; no tests were rerun for deleting the workflow invocation.

### Release note

None

### Check List (For Author)

- Behavior changed: Yes (see Details).
- Does this need documentation: No.
- Pre-submission status: Ready.

### AI Disclosure

- Tool: Codex.
- Generated: workflow simplification and this description.
- Human review: Pending author and maintainer review.
@suxiaogang223 suxiaogang223 changed the title [deps](paimon) Upgrade paimon-rust to 0.5.0 and Rust to 1.95.0 [deps](paimon) Upgrade paimon-rust to 0.5.0 Nov 16, 2026
@suxiaogang223

Copy link
Copy Markdown
Member Author

run buildall

### Background

Doris links the Paimon and Lance Rust static libraries into the BE. Master currently pins paimon-rust 0.5.0-rc1 and builds both libraries with Rust 1.94.0.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#69886 (the write integration can use this dependency after adapting to the upstream C APIs)
PR Type: Release/Dependency

Problem Summary:

Adopt the formal paimon-rust 0.5.0 release, which requires Rust 1.95.0. Build Lance with the same toolchain and reject stale installed dependencies so an existing compilation image cannot silently keep using the previous artifacts.

### Details

- **Versions:** paimon-rust 0.5.0-rc1 → 0.5.0; default/minimum Rust toolchain for Paimon and Lance 1.94.0 → 1.95.0. Pin the release archive checksum and retain locked Cargo builds.
- **Upstream changes:** The [0.5.0 release](https://github.com/apache/paimon-rust/releases/tag/v0.5.0) includes read compatibility fixes, bounded commit-message range export, pinned Arrow schema export, explicit writer close, and retained-input flush accounting. This PR consumes the upstream source without a Doris write patch.
- **Compatibility:** Both Rust static libraries need rebuilding together. Installation receipts distinguish the new release/toolchain baseline from older prebuilt artifacts. Images that ship only installed dependencies must be refreshed; the existing build guard reports this before deleting anything. Paimon resolves OpenSSL from the Doris third-party install prefix.
- **Behavior change:** Native Paimon reads use the 0.5.0 implementation; stale installed Rust artifacts require rebuilding.
- **Rollback:** Revert this dependency change and rebuild both Rust libraries together with Rust 1.94.0; replace the Paimon header and archive as a pair.

### Testing

Completed locally on Linux x86_64:

- `python3 thirdparty/test/paimon-build-test.py`: all 28 tests passed, including stale installs, failed publication, automatic toolchain selection and explicit old-toolchain rejection.
- The new tests fail against the pre-upgrade build scripts.
- Built and installed Lance with Rust 1.95.0 and `paimon-c` 0.5.0 with the same toolchain, using the existing third-party build entry point and locked Cargo dependencies.
- Linked and called both static libraries from one executable. Read the existing Paimon batch-size fixture: verified every key and 128 KiB string across all 65 rows and 22 batches. Also checked pinned schema export, serializer version 14 and commit-count error handling.
- The smoke executable has no dynamic OpenSSL dependency; the Paimon build selected OpenSSL from this worktree's third-party prefix.
- `bash -n` on changed shell files and `git diff --check`: passed.

The complete ASAN BE build is in progress. BE unit tests, external regression suites and the final production ELF compatibility audit have not been completed.

### Release note

Upgrade the native Paimon dependency to paimon-rust 0.5.0. Building Doris now selects Rust 1.95.0 for both Paimon and Lance; older prebuilt third-party installations require a rebuild or refreshed compilation image.

### Check List (For Author)

- Behavior changed: Yes (see Details).
- Does this need documentation: No new user configuration; the build requirement is described above.
- Pre-submission status: Not ready: complete BE build/CI validation is pending; native build and smoke checks passed.

### AI Disclosure

- Tool: Codex.
- Generated: dependency/build changes, build-script tests, and this description.
- Human review: Pending author and maintainer review.
### Background

The third-party workflow has a fast script-test job that runs without compiling Rust or downloading native dependencies. The Paimon build-script tests exercise the real build functions against local stubs.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#70598 (dependency upgrade)
PR Type: Test

Problem Summary:

Run the Paimon build-script suite in the existing CI script-test job so the new toolchain and installation checks continue to be verified after the upgrade.

### Details

- **Scenarios:** Old toolchains, stale installed artifacts, incomplete publication, selected Cargo, and offline metadata generation.
- **What existing tests miss:** The workflow currently exercises other dependency scripts without invoking the Paimon build-script suite.
- **Fails when broken:** The added checks fail against the previous Rust 1.94 build scripts.

### Testing

All 28 Paimon build-script tests passed locally. git diff --check passed.

### Release note

None

### Check List (For Author)

- Behavior changed: No.
- Does this need documentation: No.
- Pre-submission status: Ready; CI is pending.

### AI Disclosure

- Tool: Codex.
- Generated: workflow invocation and this description.
- Human review: Pending author and maintainer review.
### Background

The Paimon upgrade changes the release archive and moves both Rust libraries to the Rust 1.95.0 baseline. The existing build scripts already select and install the required toolchain.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#70598 (simplifies the dependency upgrade)
PR Type: Refactor

Problem Summary:

Keep this upgrade focused by removing the added installation receipts and Rust installation mock scaffolding. Upgrading the prebuilt dependencies remains an explicit paired rebuild of Paimon and Lance.

### Details

- **Moves vs. logic changes:** Remove the new receipt publication and rebuild conditions. Restore the existing test structure, update its toolchain fixture, and check the Doris OpenSSL prefix in the existing Cargo stub.
- **Behavior change:** Missing version receipts no longer force rebuilding all third-party dependencies. Existing toolchain selection and interrupted-install handling remain unchanged.
- **Compatibility:** Rebuild Paimon and Lance together with Rust 1.95.0 or refresh the compilation image.
- **Performance impact:** Avoid an automatic full dependency rebuild based on new receipt files.
- **Rollback:** Revert this commit to restore the receipt mechanism and additional tests.

### Testing

All 19 existing Paimon build-script tests passed. Shell syntax checks and git diff --check passed. The native versions and compiler inputs are unchanged from the previous successful dependency, ASAN BE/FE and Paimon write/read validations; those builds were not repeated for this simplification.

### Release note

None

### Check List (For Author)

- Behavior changed: Yes (see Details).
- Does this need documentation: No; the paired rebuild command is in the PR description.
- Pre-submission status: Partially verified: local script checks passed; CI for this revision is pending.

### AI Disclosure

- Tool: Codex.
- Generated: simplification, existing test adaptation, and this description.
- Human review: Pending author and maintainer review.
### Background

The Paimon build-script suite exists on master and can be run locally. This dependency upgrade had also added it to the third-party workflow.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#70598 (reduces upgrade scope)
PR Type: Refactor

Problem Summary:

Keep the dependency upgrade focused by restoring the workflow to master. Adding existing tests to CI can be handled separately.

### Details

- **Moves vs. logic changes:** Remove the new workflow invocation; no build script or test implementation changes.
- **Behavior change:** No dependency build behavior change. The workflow no longer adds the Paimon script suite.
- **Compatibility:** Unchanged.
- **Performance impact:** None for Doris or dependency compilation.
- **Rollback:** Restore the workflow invocation.

### Testing

Verified that the workflow matches master and git diff --check passes. The 19 script tests previously passed locally; no tests were rerun for deleting the workflow invocation.

### Release note

None

### Check List (For Author)

- Behavior changed: Yes (see Details).
- Does this need documentation: No.
- Pre-submission status: Ready.

### AI Disclosure

- Tool: Codex.
- Generated: workflow simplification and this description.
- Human review: Pending author and maintainer review.
@suxiaogang223
suxiaogang223 force-pushed the codex/paimon-rust-0.5.0 branch from b6e2079 to 51af98c Compare November 17, 2026 02:40
@suxiaogang223

Copy link
Copy Markdown
Member Author

run buildall

@suxiaogang223

Copy link
Copy Markdown
Member Author

Local review completed with Codex using Doris's code-review skill. No findings. Verified the upstream Paimon 0.5.0 archive checksum and Rust 1.95.0 requirement, checked the C signatures used by Doris, and passed all 19 Paimon build-script tests with GNU Bash 5.3. Linux, macOS, and macOS ARM64 third-party builds passed in CI for this head. Existing compilation images require the paired Paimon and Lance rebuild described in the PR.

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#70598
commit: 51af98c132633b7f2d00c873fa07f7cc807b242d
base: 687a22d2e5dd32fd2d96371b5fc01a4cd13aad56
reviewed_at: 2026-11-17T06:23:09Z
reviewer: suxiaogang223
model: gpt-6.1-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

@morningman
morningman merged commit bb13a74 into apache:master Nov 17, 2026
34 of 35 checks passed
@suxiaogang223
suxiaogang223 deleted the codex/paimon-rust-0.5.0 branch November 17, 2026 07:30
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Nov 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants