Repository navigation
[deps](paimon) Upgrade paimon-rust to 0.5.0 - #70598
Merged
Merged
Conversation
suxiaogang223
added a commit
to suxiaogang223/doris
that referenced
this pull request
Nov 16, 2026
### Background The third-party workflow has a fast script-test job that runs without compiling Rust or downloading native dependencies. The Paimon build-script tests exercise the real build functions against local stubs. ### What problem does this PR solve? Issue Number: None Related PR: apache#70598 (dependency upgrade) PR Type: Test Problem Summary: Run the Paimon build-script suite in the existing CI script-test job so the new toolchain and installation checks continue to be verified after the upgrade. ### Details - **Scenarios:** Old toolchains, stale installed artifacts, incomplete publication, selected Cargo, and offline metadata generation. - **What existing tests miss:** The workflow currently exercises other dependency scripts without invoking the Paimon build-script suite. - **Fails when broken:** The added checks fail against the previous Rust 1.94 build scripts. ### Testing All 28 Paimon build-script tests passed locally. git diff --check passed. ### Release note None ### Check List (For Author) - Behavior changed: No. - Does this need documentation: No. - Pre-submission status: Ready; CI is pending. ### AI Disclosure - Tool: Codex. - Generated: workflow invocation and this description. - Human review: Pending author and maintainer review.
suxiaogang223
marked this pull request as ready for review
November 16, 2026 04:37
suxiaogang223
requested review from
hello-stephen,
morningman and
yiguolei
as code owners
November 16, 2026 04:37
suxiaogang223
added a commit
to suxiaogang223/doris
that referenced
this pull request
Nov 16, 2026
### Background The Paimon upgrade changes the release archive and moves both Rust libraries to the Rust 1.95.0 baseline. The existing build scripts already select and install the required toolchain. ### What problem does this PR solve? Issue Number: None Related PR: apache#70598 (simplifies the dependency upgrade) PR Type: Refactor Problem Summary: Keep this upgrade focused by removing the added installation receipts and Rust installation mock scaffolding. Upgrading the prebuilt dependencies remains an explicit paired rebuild of Paimon and Lance. ### Details - **Moves vs. logic changes:** Remove the new receipt publication and rebuild conditions. Restore the existing test structure, update its toolchain fixture, and check the Doris OpenSSL prefix in the existing Cargo stub. - **Behavior change:** Missing version receipts no longer force rebuilding all third-party dependencies. Existing toolchain selection and interrupted-install handling remain unchanged. - **Compatibility:** Rebuild Paimon and Lance together with Rust 1.95.0 or refresh the compilation image. - **Performance impact:** Avoid an automatic full dependency rebuild based on new receipt files. - **Rollback:** Revert this commit to restore the receipt mechanism and additional tests. ### Testing All 19 existing Paimon build-script tests passed. Shell syntax checks and git diff --check passed. The native versions and compiler inputs are unchanged from the previous successful dependency, ASAN BE/FE and Paimon write/read validations; those builds were not repeated for this simplification. ### Release note None ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No; the paired rebuild command is in the PR description. - Pre-submission status: Partially verified: local script checks passed; CI for this revision is pending. ### AI Disclosure - Tool: Codex. - Generated: simplification, existing test adaptation, and this description. - Human review: Pending author and maintainer review.
suxiaogang223
added a commit
to suxiaogang223/doris
that referenced
this pull request
Nov 16, 2026
### Background The Paimon build-script suite exists on master and can be run locally. This dependency upgrade had also added it to the third-party workflow. ### What problem does this PR solve? Issue Number: None Related PR: apache#70598 (reduces upgrade scope) PR Type: Refactor Problem Summary: Keep the dependency upgrade focused by restoring the workflow to master. Adding existing tests to CI can be handled separately. ### Details - **Moves vs. logic changes:** Remove the new workflow invocation; no build script or test implementation changes. - **Behavior change:** No dependency build behavior change. The workflow no longer adds the Paimon script suite. - **Compatibility:** Unchanged. - **Performance impact:** None for Doris or dependency compilation. - **Rollback:** Restore the workflow invocation. ### Testing Verified that the workflow matches master and git diff --check passes. The 19 script tests previously passed locally; no tests were rerun for deleting the workflow invocation. ### Release note None ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No. - Pre-submission status: Ready. ### AI Disclosure - Tool: Codex. - Generated: workflow simplification and this description. - Human review: Pending author and maintainer review.
Member
Author
|
run buildall |
Gabriel39
approved these changes
Nov 16, 2026
### Background Doris links the Paimon and Lance Rust static libraries into the BE. Master currently pins paimon-rust 0.5.0-rc1 and builds both libraries with Rust 1.94.0. ### What problem does this PR solve? Issue Number: None Related PR: apache#69886 (the write integration can use this dependency after adapting to the upstream C APIs) PR Type: Release/Dependency Problem Summary: Adopt the formal paimon-rust 0.5.0 release, which requires Rust 1.95.0. Build Lance with the same toolchain and reject stale installed dependencies so an existing compilation image cannot silently keep using the previous artifacts. ### Details - **Versions:** paimon-rust 0.5.0-rc1 → 0.5.0; default/minimum Rust toolchain for Paimon and Lance 1.94.0 → 1.95.0. Pin the release archive checksum and retain locked Cargo builds. - **Upstream changes:** The [0.5.0 release](https://github.com/apache/paimon-rust/releases/tag/v0.5.0) includes read compatibility fixes, bounded commit-message range export, pinned Arrow schema export, explicit writer close, and retained-input flush accounting. This PR consumes the upstream source without a Doris write patch. - **Compatibility:** Both Rust static libraries need rebuilding together. Installation receipts distinguish the new release/toolchain baseline from older prebuilt artifacts. Images that ship only installed dependencies must be refreshed; the existing build guard reports this before deleting anything. Paimon resolves OpenSSL from the Doris third-party install prefix. - **Behavior change:** Native Paimon reads use the 0.5.0 implementation; stale installed Rust artifacts require rebuilding. - **Rollback:** Revert this dependency change and rebuild both Rust libraries together with Rust 1.94.0; replace the Paimon header and archive as a pair. ### Testing Completed locally on Linux x86_64: - `python3 thirdparty/test/paimon-build-test.py`: all 28 tests passed, including stale installs, failed publication, automatic toolchain selection and explicit old-toolchain rejection. - The new tests fail against the pre-upgrade build scripts. - Built and installed Lance with Rust 1.95.0 and `paimon-c` 0.5.0 with the same toolchain, using the existing third-party build entry point and locked Cargo dependencies. - Linked and called both static libraries from one executable. Read the existing Paimon batch-size fixture: verified every key and 128 KiB string across all 65 rows and 22 batches. Also checked pinned schema export, serializer version 14 and commit-count error handling. - The smoke executable has no dynamic OpenSSL dependency; the Paimon build selected OpenSSL from this worktree's third-party prefix. - `bash -n` on changed shell files and `git diff --check`: passed. The complete ASAN BE build is in progress. BE unit tests, external regression suites and the final production ELF compatibility audit have not been completed. ### Release note Upgrade the native Paimon dependency to paimon-rust 0.5.0. Building Doris now selects Rust 1.95.0 for both Paimon and Lance; older prebuilt third-party installations require a rebuild or refreshed compilation image. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No new user configuration; the build requirement is described above. - Pre-submission status: Not ready: complete BE build/CI validation is pending; native build and smoke checks passed. ### AI Disclosure - Tool: Codex. - Generated: dependency/build changes, build-script tests, and this description. - Human review: Pending author and maintainer review.
### Background The third-party workflow has a fast script-test job that runs without compiling Rust or downloading native dependencies. The Paimon build-script tests exercise the real build functions against local stubs. ### What problem does this PR solve? Issue Number: None Related PR: apache#70598 (dependency upgrade) PR Type: Test Problem Summary: Run the Paimon build-script suite in the existing CI script-test job so the new toolchain and installation checks continue to be verified after the upgrade. ### Details - **Scenarios:** Old toolchains, stale installed artifacts, incomplete publication, selected Cargo, and offline metadata generation. - **What existing tests miss:** The workflow currently exercises other dependency scripts without invoking the Paimon build-script suite. - **Fails when broken:** The added checks fail against the previous Rust 1.94 build scripts. ### Testing All 28 Paimon build-script tests passed locally. git diff --check passed. ### Release note None ### Check List (For Author) - Behavior changed: No. - Does this need documentation: No. - Pre-submission status: Ready; CI is pending. ### AI Disclosure - Tool: Codex. - Generated: workflow invocation and this description. - Human review: Pending author and maintainer review.
### Background The Paimon upgrade changes the release archive and moves both Rust libraries to the Rust 1.95.0 baseline. The existing build scripts already select and install the required toolchain. ### What problem does this PR solve? Issue Number: None Related PR: apache#70598 (simplifies the dependency upgrade) PR Type: Refactor Problem Summary: Keep this upgrade focused by removing the added installation receipts and Rust installation mock scaffolding. Upgrading the prebuilt dependencies remains an explicit paired rebuild of Paimon and Lance. ### Details - **Moves vs. logic changes:** Remove the new receipt publication and rebuild conditions. Restore the existing test structure, update its toolchain fixture, and check the Doris OpenSSL prefix in the existing Cargo stub. - **Behavior change:** Missing version receipts no longer force rebuilding all third-party dependencies. Existing toolchain selection and interrupted-install handling remain unchanged. - **Compatibility:** Rebuild Paimon and Lance together with Rust 1.95.0 or refresh the compilation image. - **Performance impact:** Avoid an automatic full dependency rebuild based on new receipt files. - **Rollback:** Revert this commit to restore the receipt mechanism and additional tests. ### Testing All 19 existing Paimon build-script tests passed. Shell syntax checks and git diff --check passed. The native versions and compiler inputs are unchanged from the previous successful dependency, ASAN BE/FE and Paimon write/read validations; those builds were not repeated for this simplification. ### Release note None ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No; the paired rebuild command is in the PR description. - Pre-submission status: Partially verified: local script checks passed; CI for this revision is pending. ### AI Disclosure - Tool: Codex. - Generated: simplification, existing test adaptation, and this description. - Human review: Pending author and maintainer review.
### Background The Paimon build-script suite exists on master and can be run locally. This dependency upgrade had also added it to the third-party workflow. ### What problem does this PR solve? Issue Number: None Related PR: apache#70598 (reduces upgrade scope) PR Type: Refactor Problem Summary: Keep the dependency upgrade focused by restoring the workflow to master. Adding existing tests to CI can be handled separately. ### Details - **Moves vs. logic changes:** Remove the new workflow invocation; no build script or test implementation changes. - **Behavior change:** No dependency build behavior change. The workflow no longer adds the Paimon script suite. - **Compatibility:** Unchanged. - **Performance impact:** None for Doris or dependency compilation. - **Rollback:** Restore the workflow invocation. ### Testing Verified that the workflow matches master and git diff --check passes. The 19 script tests previously passed locally; no tests were rerun for deleting the workflow invocation. ### Release note None ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No. - Pre-submission status: Ready. ### AI Disclosure - Tool: Codex. - Generated: workflow simplification and this description. - Human review: Pending author and maintainer review.
suxiaogang223
force-pushed
the
codex/paimon-rust-0.5.0
branch
from
November 17, 2026 02:40
b6e2079 to
51af98c
Compare
Member
Author
|
run buildall |
Member
Author
|
Local review completed with Codex using Doris's schema: doris-repo-review/v1
status: PASS
pr: apache/doris#70598
commit: 51af98c132633b7f2d00c873fa07f7cc807b242d
base: 687a22d2e5dd32fd2d96371b5fc01a4cd13aad56
reviewed_at: 2026-11-17T06:23:09Z
reviewer: suxiaogang223
model: gpt-6.1-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true |
morningman
approved these changes
Nov 17, 2026
Contributor
|
PR approved by anyone and no changes requested. |
Contributor
|
PR approved by at least one committer and no changes requested. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
Doris uses paimon-rust for native Paimon reads. Paimon and Lance are linked into the BE as Rust static libraries.
What problem does this PR solve?
Issue Number: None
Related PR: #69886 (Rust write integration)
PR Type: Release/Dependency
Problem Summary:
Upgrade to the official paimon-rust 0.5.0 release. Update both libraries to its required Rust 1.95.0 baseline and use Doris's OpenSSL prefix when building Paimon.
Details
Testing
Passed locally on Linux x86_64:
Release note
Upgrade paimon-rust to 0.5.0. Building Paimon and Lance requires Rust 1.95.0 or newer, using the same toolchain for both.
Check List (For Author)
AI Disclosure