Skip to content

[feature](paimon) Support writes using paimon-rust - #69886

Open
suxiaogang223 wants to merge 7 commits into
apache:masterfrom
suxiaogang223:codex/branch-4.2-paimon-rust-write
Open

suxiaogang223 wants to merge 7 commits into
apache:masterfrom
suxiaogang223:codex/branch-4.2-paimon-rust-write

Conversation

@suxiaogang223

@suxiaogang223 suxiaogang223 commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Background

Paimon writes currently use the Java writer through JNI. FE collects BE commit messages and publishes the Paimon snapshot.

What problem does this PR solve?

Issue Number: #65086
Related PR: None
PR Type: Feature

Problem Summary:

Add Paimon append writes through the Rust C FFI. Prefer Rust for supported writes and select JNI before consuming input for unsupported cases. Runtime failures fail the statement.

Details

  • Change map: About 1.9k changed production lines, 2.1k test lines, and 34 protocol lines. Covers writer integration, a shared table descriptor, resource accounting, transaction diagnostics, and tests.
  • Scope: Main-branch, unpartitioned append-only Parquet tables with bucket=-1, supported scalar types, and local or explicitly configured S3 storage.
  • Design: Share a pinned table descriptor across Rust reads and Rust/JNI writes. BE sends Java v14 commit messages; FE uses the existing transaction framework. Confirmed aborts clean up files; uncertain report or commit outcomes preserve them.
  • Usage: INSERT INTO; set enable_paimon_rust_writer=false to force JNI.
  • Default behavior: Rust is preferred for supported writes. Per-writer estimated memory is 64 MiB; concurrency is limited to 8 writers per BE. Both paimon_rust_writer_memory_limit_bytes and paimon_rust_writer_max_concurrency can be changed online through /api/update_config for new writers, without restarting Doris. After a confirmed uncommitted failure, adjust limits and retry. Check unknown commit outcomes before retrying to avoid duplicate appends.
  • Compatibility: Unsupported writes select JNI. Incompatible table descriptors are rejected. Connector plugins use SPI API 33.0. Accounting is estimated; it does not enforce a complete Rust allocator or RSS limit, and does not provide spill or memory revocation.
  • Behavior change: Supported append writes use Rust by default.

BE writes to the existing catalog storage endpoint. INSERT requires the existing table LOAD privilege; catalog creation and configuration retain their existing privilege checks.

Testing

Local verification passed:

  • ASAN BE and FE builds, Java plugin packaging, Checkstyle, clang-format 16, clang-tidy, and header hygiene.
  • 57 BE tests, including 21 FFI writer tests; 1,175 FE/Java tests with one skipped and no failures or errors.
  • Append writes and Rust/JNI/Spark reads; online memory adjustment and retry; memory release, cancellation, and Java v14 range framing.
  • Two-BE writes and fault injection: partial BE failure, FE failure before publication, reconciliation after publication, and uncertain outcomes with file preservation.

Release note

Support Paimon append writes using paimon-rust, preferred by default for supported tables. Writer memory and concurrency settings can be adjusted online for subsequent writers without restarting Doris.

Check List (For Author)

  • Behavior changed: Yes (see Details).
  • Does this need documentation: Yes. Document the supported tables, writer switch, and online limits in the user documentation.
  • Pre-submission status: Ready for review; local verification passed. CI pending.

AI Disclosure

  • Tool: Codex (model ID unavailable in this session).
  • Generated: Implementation, tests, and this description.
  • Human review: Current adaptation awaiting author review.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Oct 30, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69886

Problem Summary: When one BE fails, FE can roll back and unregister the query
before a successful BE reports its prepared Paimon files. The late report is
then discarded, leaving uncommitted files behind. This is reproducible with
two BEs by delaying the successful writer's final report.

Make rollback terminal and atomically hand off reported files under the
transaction lock. Retain bounded contexts for recently aborted inserts so
reports received after query removal, or racing with transaction removal,
can still abort their files. Release payload references and perform storage
I/O outside the lock. Never register committing, committed, or unknown-outcome
transactions for late file deletion.

Add fault injection and distributed regression coverage for BE prepare
failure, delayed reports, failure before FE commit, errors after snapshot
publication, and unknown commit outcomes. Process crashes before reporting
files still require orphan-file maintenance.

### Release note

Clean up uncommitted Paimon files when a successful BE reports after another
BE has failed and FE has rolled back the INSERT.

### Check List (For Author)

- Test: Regression test / Unit Test / Manual test
    - 10 Rust BE tests pass under ASAN; 31 focused FE tests pass.
    - Distributed failure suite passes on both FE distribution planners;
      expected output generated by the standard runner and verified normally.
    - Existing Paimon Rust append regression passes.
    - Manual two-BE SIGKILL test confirms failed INSERT publishes no snapshot,
      cleans the healthy writer's files, and succeeds after BE restart.
    - Full ASAN BE/FE build, updated FE build with Checkstyle, and clang-format
      16 preset check pass. Clang-tidy is blocked by existing NOLINTEND and
      local system-header parsing errors.
- Behavior changed: Yes, late reports for aborted Paimon inserts clean files
- Does this need documentation: No, internal failure cleanup
@suxiaogang223 suxiaogang223 changed the title [feature](be) Add opt-in Paimon Rust append writes [feature](be) Add Paimon Rust append writes with shared table metadata Oct 30, 2026
@suxiaogang223 suxiaogang223 changed the title [feature](be) Add Paimon Rust append writes with shared table metadata [feature](paimon) Support writes using paimon-rust Nov 2, 2026
@suxiaogang223

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 82.92% (403/486) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 64.81% (30368/46860)
Line Coverage 49.66% (321327/646990)
Region Coverage 45.46% (259080/569878)
Branch Coverage 47.10% (122358/259795)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 82.89% (402/485) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.41% (34775/45511)
Line Coverage 61.25% (392983/641599)
Region Coverage 58.07% (331451/570735)
Branch Coverage 58.99% (153067/259483)

@suxiaogang223
suxiaogang223 force-pushed the codex/branch-4.2-paimon-rust-write branch from 9ae32af to c489854 Compare November 3, 2026 08:21
@suxiaogang223
suxiaogang223 changed the base branch from branch-4.2 to master November 3, 2026 08:22
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Nov 3, 2026
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: A pipeline can finish early or be cancelled before its
sink opens. PaimonTableWriter.close must return the incoming status in
that case, because neither its backend nor its profile has been created.
Guard the whole close path instead of only its final profile update.
Add coverage for successful early completion alongside cancellation.

### Release note

Avoid a Paimon writer crash when a pipeline finishes before opening its sink.

### Check List (For Author)

- Test: Added early-completion unit coverage; clang-format 16 and BE build
  hygiene passed. Native compilation and tests have not been run on master.
- Behavior changed: Yes. Pre-open completion preserves its original status.
- Does this need documentation: No.
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Nov 3, 2026
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: The master plugin transaction manager removes a transaction
before invoking the connector commit. A failure before snapshot publication
therefore cannot rely on the engine's later rollback to clean up acknowledged
BE files. Roll back within the Paimon commit failure path while still PREPARED;
leave COMMITTING/OUTCOME_UNKNOWN files untouched. Test both outcomes with real
prepared data files and verify that subsequent rollback does not abort twice.

Add an exact license-check exclusion for the Rust-generated base64 protocol
fixture, following existing golden-file exclusions; text headers would change
its decoded bytes. Remove an obsolete comment describing FFI as future work.

### Release note

Clean up unpublished Paimon data files when FE fails before attempting commit.

### Check List (For Author)

- Test: FE and Java plugin build with Checkstyle passed. Paimon connector
  904 tests, no failures/errors, one skipped, including the real-file cleanup
  and unknown-outcome tests. clang-format 16, BE build hygiene, shell syntax
  and license YAML validation passed. BE compilation and remaining FE/JNI
  tests are running; native and SQL validation are not claimed as passed.
- Behavior changed: Yes. Pre-publication commit failures clean up immediately;
  uncertain publication outcomes still preserve files.
- Does this need documentation: No.
@suxiaogang223

Copy link
Copy Markdown
Member Author

run buildall

suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Nov 3, 2026
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: Use the existing JNI Thrift serialization helper for the
shared Paimon table descriptor, including its length and exception checks.
Separate Java open arguments and cached Rust table initialization from their
caller functions, and shorten repetitive comments so modified functions meet
clang-tidy complexity and size limits. Apply pointer qualification and
explicit aggregate member initialization reported by clang-tidy.

### Release note

None

### Check List (For Author)

- Test: clang-format 16, BE build hygiene and changed-line clang-tidy passed
  for the affected writer/descriptor/reader paths; JNI source also passed
  full-file clang-tidy. FE/Java build and targeted tests passed before this
  C++ refactor. Final master ASAN BE build is running; native/SQL tests are
  not claimed as passed.
- Behavior changed: No. Reuse existing checked JNI serialization.
- Does this need documentation: No.
@suxiaogang223

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 23.81% (5/21) 🎉
Increment coverage report
Complete coverage report

@suxiaogang223
suxiaogang223 force-pushed the codex/branch-4.2-paimon-rust-write branch from d2f5e2b to 496da98 Compare November 4, 2026 03:03
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Nov 4, 2026
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: A pipeline can finish early or be cancelled before its
sink opens. PaimonTableWriter.close must return the incoming status in
that case, because neither its backend nor its profile has been created.
Guard the whole close path instead of only its final profile update.
Add coverage for successful early completion alongside cancellation.

### Release note

Avoid a Paimon writer crash when a pipeline finishes before opening its sink.

### Check List (For Author)

- Test: Added early-completion unit coverage; clang-format 16 and BE build
  hygiene passed. Native compilation and tests have not been run on master.
- Behavior changed: Yes. Pre-open completion preserves its original status.
- Does this need documentation: No.
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Nov 4, 2026
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: The master plugin transaction manager removes a transaction
before invoking the connector commit. A failure before snapshot publication
therefore cannot rely on the engine's later rollback to clean up acknowledged
BE files. Roll back within the Paimon commit failure path while still PREPARED;
leave COMMITTING/OUTCOME_UNKNOWN files untouched. Test both outcomes with real
prepared data files and verify that subsequent rollback does not abort twice.

Add an exact license-check exclusion for the Rust-generated base64 protocol
fixture, following existing golden-file exclusions; text headers would change
its decoded bytes. Remove an obsolete comment describing FFI as future work.

### Release note

Clean up unpublished Paimon data files when FE fails before attempting commit.

### Check List (For Author)

- Test: FE and Java plugin build with Checkstyle passed. Paimon connector
  904 tests, no failures/errors, one skipped, including the real-file cleanup
  and unknown-outcome tests. clang-format 16, BE build hygiene, shell syntax
  and license YAML validation passed. BE compilation and remaining FE/JNI
  tests are running; native and SQL validation are not claimed as passed.
- Behavior changed: Yes. Pre-publication commit failures clean up immediately;
  uncertain publication outcomes still preserve files.
- Does this need documentation: No.
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Nov 4, 2026
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: Use the existing JNI Thrift serialization helper for the
shared Paimon table descriptor, including its length and exception checks.
Separate Java open arguments and cached Rust table initialization from their
caller functions, and shorten repetitive comments so modified functions meet
clang-tidy complexity and size limits. Apply pointer qualification and
explicit aggregate member initialization reported by clang-tidy.

### Release note

None

### Check List (For Author)

- Test: clang-format 16, BE build hygiene and changed-line clang-tidy passed
  for the affected writer/descriptor/reader paths; JNI source also passed
  full-file clang-tidy. FE/Java build and targeted tests passed before this
  C++ refactor. Final master ASAN BE build is running; native/SQL tests are
  not claimed as passed.
- Behavior changed: No. Reuse existing checked JNI serialization.
- Does this need documentation: No.
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Nov 4, 2026
### What problem does this PR solve?

Related PR: apache#69886, apache#70075, apache#69343

Problem Summary: Master now provides paimon-rust v0.5.0-rc1 and Java Paimon
2.1.0. Rebase the writer on master instead of carrying separate dependency
upgrades. Reuse the upstream Java v14 serializer with bounded accumulated
output, remove the v11 and legacy DataFileMeta encoding patch, and regenerate
the real two-writer interop fixture. Reject old protocol requests before
writer admission. Preserve master's vector-search field 42 and use optional
field 43 for the shared descriptor; retain master's SPI additions and bump
the write diagnostics surface to API 30.

The previous BE UT crash was caused by a validation test bypassing init()
and reading an uninitialized scan-params pointer. Explicitly initialize its
missing-descriptor input.

### Release note

Rust Paimon writes use the Java v14 commit protocol matching master's Paimon
2.1.0 SDK and inherit master's paimon-rust dependency version.

### Check List (For Author)

- Test: Third-party build with Rust 1.94.0; real C FFI append, codec bounds,
  version rejection, message ranges and resource release; FE/Java build with
  Checkstyle; Paimon connector 904 tests (one skip); connector SPI 152 tests;
  six Paimon JNI static layout checks; Thrift generation/field audit;
  clang-format 16, header hygiene, shell syntax and whitespace checks.
  JNI Paimon 186 and FE integration 36 tests passed. BE compilation, native tests,
  clang-tidy against the new build and SQL regression are deferred at the
  user's request to leave resources for another PR.
- Behavior changed: Yes; Rust commit messages now use v14 instead of v11.
- Does this need documentation: Yes; existing feature documentation remains
  pending with the draft PR.
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Nov 4, 2026
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: The upgraded paimon-rust writer needs native coverage of
credit release, cleanup and the v14 transport. Use the same Parquet buffer
option as the production descriptor instead of the generic write-buffer-size
option, which does not control the Parquet buffer.

Cover empty writes, prepare failure after rolling a file, and sustained
32 MiB dictionary input under a 16 MiB SDK input budget without rolling the
file. Extend the memory-failure case to retry the same batch with a larger
runtime limit. Check complete big-endian frame fields and the metadata
credit retained until report acknowledgement. The exercised paths required
no production logic changes.

### Release note

None

### Check List (For Author)

- Test: ASAN BE compilation and packaging with build.sh; 164 related BE
  tests passed through run-be-ut.sh (20 FFI writer tests, no failures or
  skips); clang-tidy for five modified production translation units and
  the full updated FFI test file; clang-format 16, header hygiene and
  whitespace checks. SQL/distributed regression was not rerun.
- Behavior changed: No; test-only changes.
- Does this need documentation: No; test-only changes.
@suxiaogang223

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.65% (2101/2741)
Line Coverage 65.99% (38786/58772)
Region Coverage 53.76% (36417/67743)
Branch Coverage 57.27% (11868/20722)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 23.81% (5/21) 🎉
Increment coverage report
Complete coverage report

@suxiaogang223

Copy link
Copy Markdown
Member Author

run beut

@suxiaogang223
suxiaogang223 force-pushed the codex/branch-4.2-paimon-rust-write branch from 8cf2388 to 4078a16 Compare November 5, 2026 05:06
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Nov 5, 2026
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: A pipeline can finish early or be cancelled before its
sink opens. PaimonTableWriter.close must return the incoming status in
that case, because neither its backend nor its profile has been created.
Guard the whole close path instead of only its final profile update.
Add coverage for successful early completion alongside cancellation.

### Release note

Avoid a Paimon writer crash when a pipeline finishes before opening its sink.

### Check List (For Author)

- Test: Added early-completion unit coverage; clang-format 16 and BE build
  hygiene passed. Native compilation and tests have not been run on master.
- Behavior changed: Yes. Pre-open completion preserves its original status.
- Does this need documentation: No.
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Nov 5, 2026
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: The master plugin transaction manager removes a transaction
before invoking the connector commit. A failure before snapshot publication
therefore cannot rely on the engine's later rollback to clean up acknowledged
BE files. Roll back within the Paimon commit failure path while still PREPARED;
leave COMMITTING/OUTCOME_UNKNOWN files untouched. Test both outcomes with real
prepared data files and verify that subsequent rollback does not abort twice.

Add an exact license-check exclusion for the Rust-generated base64 protocol
fixture, following existing golden-file exclusions; text headers would change
its decoded bytes. Remove an obsolete comment describing FFI as future work.

### Release note

Clean up unpublished Paimon data files when FE fails before attempting commit.

### Check List (For Author)

- Test: FE and Java plugin build with Checkstyle passed. Paimon connector
  904 tests, no failures/errors, one skipped, including the real-file cleanup
  and unknown-outcome tests. clang-format 16, BE build hygiene, shell syntax
  and license YAML validation passed. BE compilation and remaining FE/JNI
  tests are running; native and SQL validation are not claimed as passed.
- Behavior changed: Yes. Pre-publication commit failures clean up immediately;
  uncertain publication outcomes still preserve files.
- Does this need documentation: No.
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Nov 5, 2026
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: Use the existing JNI Thrift serialization helper for the
shared Paimon table descriptor, including its length and exception checks.
Separate Java open arguments and cached Rust table initialization from their
caller functions, and shorten repetitive comments so modified functions meet
clang-tidy complexity and size limits. Apply pointer qualification and
explicit aggregate member initialization reported by clang-tidy.

### Release note

None

### Check List (For Author)

- Test: clang-format 16, BE build hygiene and changed-line clang-tidy passed
  for the affected writer/descriptor/reader paths; JNI source also passed
  full-file clang-tidy. FE/Java build and targeted tests passed before this
  C++ refactor. Final master ASAN BE build is running; native/SQL tests are
  not claimed as passed.
- Behavior changed: No. Reuse existing checked JNI serialization.
- Does this need documentation: No.
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Nov 5, 2026
### What problem does this PR solve?

Related PR: apache#69886, apache#70075, apache#69343

Problem Summary: Master now provides paimon-rust v0.5.0-rc1 and Java Paimon
2.1.0. Rebase the writer on master instead of carrying separate dependency
upgrades. Reuse the upstream Java v14 serializer with bounded accumulated
output, remove the v11 and legacy DataFileMeta encoding patch, and regenerate
the real two-writer interop fixture. Reject old protocol requests before
writer admission. Preserve master's vector-search field 42 and use optional
field 43 for the shared descriptor; retain master's SPI additions and bump
the write diagnostics surface to API 30.

The previous BE UT crash was caused by a validation test bypassing init()
and reading an uninitialized scan-params pointer. Explicitly initialize its
missing-descriptor input.

### Release note

Rust Paimon writes use the Java v14 commit protocol matching master's Paimon
2.1.0 SDK and inherit master's paimon-rust dependency version.

### Check List (For Author)

- Test: Third-party build with Rust 1.94.0; real C FFI append, codec bounds,
  version rejection, message ranges and resource release; FE/Java build with
  Checkstyle; Paimon connector 904 tests (one skip); connector SPI 152 tests;
  six Paimon JNI static layout checks; Thrift generation/field audit;
  clang-format 16, header hygiene, shell syntax and whitespace checks.
  JNI Paimon 186 and FE integration 36 tests passed. BE compilation, native tests,
  clang-tidy against the new build and SQL regression are deferred at the
  user's request to leave resources for another PR.
- Behavior changed: Yes; Rust commit messages now use v14 instead of v11.
- Does this need documentation: Yes; existing feature documentation remains
  pending with the draft PR.
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Nov 5, 2026
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: The upgraded paimon-rust writer needs native coverage of
credit release, cleanup and the v14 transport. Use the same Parquet buffer
option as the production descriptor instead of the generic write-buffer-size
option, which does not control the Parquet buffer.

Cover empty writes, prepare failure after rolling a file, and sustained
32 MiB dictionary input under a 16 MiB SDK input budget without rolling the
file. Extend the memory-failure case to retry the same batch with a larger
runtime limit. Check complete big-endian frame fields and the metadata
credit retained until report acknowledgement. The exercised paths required
no production logic changes.

### Release note

None

### Check List (For Author)

- Test: ASAN BE compilation and packaging with build.sh; 164 related BE
  tests passed through run-be-ut.sh (20 FFI writer tests, no failures or
  skips); clang-tidy for five modified production translation units and
  the full updated FFI test file; clang-format 16, header hygiene and
  whitespace checks. SQL/distributed regression was not rerun.
- Behavior changed: No; test-only changes.
- Does this need documentation: No; test-only changes.
@suxiaogang223

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.65% (2101/2741)
Line Coverage 66.02% (38800/58772)
Region Coverage 53.77% (36425/67743)
Branch Coverage 57.19% (11851/20722)

@suxiaogang223

Copy link
Copy Markdown
Member Author

Local review with the Doris code-review skill: PASS for 4078a16c9b67a4467275f2dfc31e125eba4a42d1 against 913532ce1673595cd552bd6f582e5b2640f45dd9. No unresolved findings.

Reviewed the Rust/JNI selection, shared pinned descriptor, v14 commit transport, memory admission/release, report ownership and FE commit/rollback flow. The six feature patches are unchanged by the rebase.

Current-head CI compilation, FE UT and Cloud UT have passed. BE UT and regression jobs are still running. This review does not mark unfinished tests as passed. Earlier local validation covered 164 related BE tests (20 FFI writer tests) and 1,541 FE/Java tests (one skip); those were not rerun during this review. Local formatting with clang-format 16, build hygiene, shell syntax and cross-release Thrift checks passed.

Doris skill checkpoints
Checkpoint Conclusion
Goal Supported append writes prefer Rust; unsupported writes select JNI before consuming input.
Scope and reuse Changes reuse the existing transaction/report framework and pinned table schema.
Concurrency and locks Per-task writer calls are serialized; admission locks exclude SDK work and group refresh; abort I/O runs outside the connector state lock.
Lifetime and static initialization Native work stops before writer credit/slots are returned; report callbacks own their metadata and tracker; metrics initialize lazily.
Online settings Mutable limits apply to new admissions; active budgets retain their original value.
Compatibility Missing descriptors, unsupported commit versions and JNI ABI mismatches return errors, consistent with the agreed mixed-version scope.
Thrift/Proto Checked master and branches 3.1/4.0/4.1/4.2: optional additions, stable meanings and no ID collisions. No protobuf changes.
Parallel paths Descriptor sharing covers Rust reads and Rust/JNI writes; SPI forwarding and S3/MinIO aliases are complete.
Conditions Backend eligibility and cleanup states correspond to concrete supported/failed paths; runtime errors fail the statement.
Coverage Tests cover budgets, release, cancellation, prepare/report failures, protocol bytes and FE outcomes; current CI remains in progress.
Test results Real writer fixtures verify Java interoperability; generated regression outputs were not edited.
Observability Profiles/metrics cover selection, resources, input, timing, failures and transaction states.
Transactions and persistence FE alone publishes snapshots; pre-publication failures clean up, uncertain outcomes retry/reconcile and preserve files. No new EditLog.
Data writes and crashes BE only prepares; ACK/rejection/ambiguous ownership is handled explicitly. Crash/cleanup-I/O orphan maintenance remains an accepted limitation.
FE/BE propagation Session forwarding, sink/scan descriptors and commit version agree; JNI rebuilds the pinned table.
Performance and memory Input/commit payloads are bounded; query/native credits are estimates. Full RSS limits, spill and revocation remain outside scope.
Other integration Checked current Coordinator/profile lifetime, transaction-manager behavior, SPI version and plugin dependencies; master includes the common BE UT build fix #70209.
schema: doris-repo-review/v1
status: PASS
pr: apache/doris#69886
commit: 4078a16c9b67a4467275f2dfc31e125eba4a42d1
base: 913532ce1673595cd552bd6f582e5b2640f45dd9
reviewed_at: 2026-11-05T07:37:01+00:00
reviewer: suxiaogang223
model: gpt-6.1-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 93.94% (31/33) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 67.44% (32207/47758)
Line Coverage 52.28% (339050/648569)
Region Coverage 47.76% (275491/576810)
Branch Coverage 49.60% (129904/261886)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 93.75% (30/32) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.65% (35928/46271)
Line Coverage 62.98% (404048/641511)
Region Coverage 59.48% (341489/574123)
Branch Coverage 60.67% (158318/260946)

suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Nov 17, 2026
### Background

Doris links the Paimon and Lance Rust static libraries into the BE. Master currently pins paimon-rust 0.5.0-rc1 and builds both libraries with Rust 1.94.0.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#69886 (the write integration can use this dependency after adapting to the upstream C APIs)
PR Type: Release/Dependency

Problem Summary:

Adopt the formal paimon-rust 0.5.0 release, which requires Rust 1.95.0. Build Lance with the same toolchain and reject stale installed dependencies so an existing compilation image cannot silently keep using the previous artifacts.

### Details

- **Versions:** paimon-rust 0.5.0-rc1 → 0.5.0; default/minimum Rust toolchain for Paimon and Lance 1.94.0 → 1.95.0. Pin the release archive checksum and retain locked Cargo builds.
- **Upstream changes:** The [0.5.0 release](https://github.com/apache/paimon-rust/releases/tag/v0.5.0) includes read compatibility fixes, bounded commit-message range export, pinned Arrow schema export, explicit writer close, and retained-input flush accounting. This PR consumes the upstream source without a Doris write patch.
- **Compatibility:** Both Rust static libraries need rebuilding together. Installation receipts distinguish the new release/toolchain baseline from older prebuilt artifacts. Images that ship only installed dependencies must be refreshed; the existing build guard reports this before deleting anything. Paimon resolves OpenSSL from the Doris third-party install prefix.
- **Behavior change:** Native Paimon reads use the 0.5.0 implementation; stale installed Rust artifacts require rebuilding.
- **Rollback:** Revert this dependency change and rebuild both Rust libraries together with Rust 1.94.0; replace the Paimon header and archive as a pair.

### Testing

Completed locally on Linux x86_64:

- `python3 thirdparty/test/paimon-build-test.py`: all 28 tests passed, including stale installs, failed publication, automatic toolchain selection and explicit old-toolchain rejection.
- The new tests fail against the pre-upgrade build scripts.
- Built and installed Lance with Rust 1.95.0 and `paimon-c` 0.5.0 with the same toolchain, using the existing third-party build entry point and locked Cargo dependencies.
- Linked and called both static libraries from one executable. Read the existing Paimon batch-size fixture: verified every key and 128 KiB string across all 65 rows and 22 batches. Also checked pinned schema export, serializer version 14 and commit-count error handling.
- The smoke executable has no dynamic OpenSSL dependency; the Paimon build selected OpenSSL from this worktree's third-party prefix.
- `bash -n` on changed shell files and `git diff --check`: passed.

The complete ASAN BE build is in progress. BE unit tests, external regression suites and the final production ELF compatibility audit have not been completed.

### Release note

Upgrade the native Paimon dependency to paimon-rust 0.5.0. Building Doris now selects Rust 1.95.0 for both Paimon and Lance; older prebuilt third-party installations require a rebuild or refreshed compilation image.

### Check List (For Author)

- Behavior changed: Yes (see Details).
- Does this need documentation: No new user configuration; the build requirement is described above.
- Pre-submission status: Not ready: complete BE build/CI validation is pending; native build and smoke checks passed.

### AI Disclosure

- Tool: Codex.
- Generated: dependency/build changes, build-script tests, and this description.
- Human review: Pending author and maintainer review.
morningman pushed a commit that referenced this pull request Nov 17, 2026
### Background

Doris uses paimon-rust for native Paimon reads. Paimon and Lance are
linked into the BE as Rust static libraries.

### What problem does this PR solve?

Issue Number: None
Related PR: #69886 (Rust write integration)
PR Type: Release/Dependency

Problem Summary:

Upgrade to the official paimon-rust 0.5.0 release. Update both libraries
to its required Rust 1.95.0 baseline and use Doris's OpenSSL prefix when
building Paimon.

### Details

- **Versions:** paimon-rust 0.5.0-rc1 → 0.5.0; Rust 1.94.0 → 1.95.0.
- **Upstream changes:** Adopt the official release without local
patches.
- **Compatibility:** Rebuild Paimon and Lance together, or refresh the
compilation image.
- **Rollback:** Revert and rebuild both libraries with Rust 1.94.0.
- **Behavior change:** Native Paimon reads use version 0.5.0.

### Testing

Passed locally on Linux x86_64:

- Paimon/Lance third-party builds and Doris ASAN BE/FE builds, including
required Java plugins.
- All 19 existing Paimon build-script tests.
- Paimon compatibility and basic write/read cases: append, partitioned
writes, empty writes and primary-key overwrite. Writes use JNI; Profile
confirms Rust reader execution.

### Release note

Upgrade paimon-rust to 0.5.0. Building Paimon and Lance requires Rust
1.95.0 or newer, using the same toolchain for both.

### Check List (For Author)

- Behavior changed: Yes (see Details).
- Does this need documentation: No.
- Pre-submission status: Partially verified: local checks passed; CI
pending.

### AI Disclosure

- Tool: Codex.
- Generated: build changes, test adaptation and PR description.
- Human review: Pending author and maintainer review.
Related PR: apache#67395

Problem Summary:

Port the branch-4.2 Rust append writer to master's connector framework and
Paimon Java SDK 2.1.0. Prefer Rust for supported writes, share a pinned
Thrift table descriptor across Rust reads and Rust/JNI writes, and retain
online writer memory/concurrency settings. Use the existing acknowledged
file-report protocol for ownership and cleanup, with a separate estimated
credit for native commit metadata until reporting completes. Expose
transaction diagnostics through connector-neutral SPI hooks (API 29.0).

Keep Java v11 commit encoding in the released paimon-rust 0.4.0 patch;
verify that SDK 2.1.0 decodes it. Preserve Rust 1.94.0. Fix closing a writer
before open and reserve Fluss field 41 by assigning Paimon scan field 42.

Support Paimon append writes using paimon-rust, enabled by default for
supported targets with JNI selection for unsupported writes. Runtime
failures fail the statement. Writer limits can be changed online for new
writers; users can tune limits and retry confirmed uncommitted failures.

- Test: FE/Java compilation and packaging with Checkstyle; Paimon connector
  902 tests (one skipped); SPI 152; JNI Paimon 186; FE integration 36;
  S3/MinIO properties 37; plugin layout; clang-format 16; BE header hygiene.
  Master BE compilation/native tests and SQL regression remain pending.
- Behavior changed: Yes. Supported append writes prefer Rust; table
  descriptions and transaction diagnostics use master's connector model.
- Does this need documentation: Yes. User documentation is pending.
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: A pipeline can finish early or be cancelled before its
sink opens. PaimonTableWriter.close must return the incoming status in
that case, because neither its backend nor its profile has been created.
Guard the whole close path instead of only its final profile update.
Add coverage for successful early completion alongside cancellation.

### Release note

Avoid a Paimon writer crash when a pipeline finishes before opening its sink.

### Check List (For Author)

- Test: Added early-completion unit coverage; clang-format 16 and BE build
  hygiene passed. Native compilation and tests have not been run on master.
- Behavior changed: Yes. Pre-open completion preserves its original status.
- Does this need documentation: No.
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: The master plugin transaction manager removes a transaction
before invoking the connector commit. A failure before snapshot publication
therefore cannot rely on the engine's later rollback to clean up acknowledged
BE files. Roll back within the Paimon commit failure path while still PREPARED;
leave COMMITTING/OUTCOME_UNKNOWN files untouched. Test both outcomes with real
prepared data files and verify that subsequent rollback does not abort twice.

Add an exact license-check exclusion for the Rust-generated base64 protocol
fixture, following existing golden-file exclusions; text headers would change
its decoded bytes. Remove an obsolete comment describing FFI as future work.

### Release note

Clean up unpublished Paimon data files when FE fails before attempting commit.

### Check List (For Author)

- Test: FE and Java plugin build with Checkstyle passed. Paimon connector
  904 tests, no failures/errors, one skipped, including the real-file cleanup
  and unknown-outcome tests. clang-format 16, BE build hygiene, shell syntax
  and license YAML validation passed. BE compilation and remaining FE/JNI
  tests are running; native and SQL validation are not claimed as passed.
- Behavior changed: Yes. Pre-publication commit failures clean up immediately;
  uncertain publication outcomes still preserve files.
- Does this need documentation: No.
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: Use the existing JNI Thrift serialization helper for the
shared Paimon table descriptor, including its length and exception checks.
Separate Java open arguments and cached Rust table initialization from their
caller functions, and shorten repetitive comments so modified functions meet
clang-tidy complexity and size limits. Apply pointer qualification and
explicit aggregate member initialization reported by clang-tidy.

### Release note

None

### Check List (For Author)

- Test: clang-format 16, BE build hygiene and changed-line clang-tidy passed
  for the affected writer/descriptor/reader paths; JNI source also passed
  full-file clang-tidy. FE/Java build and targeted tests passed before this
  C++ refactor. Final master ASAN BE build is running; native/SQL tests are
  not claimed as passed.
- Behavior changed: No. Reuse existing checked JNI serialization.
- Does this need documentation: No.
### What problem does this PR solve?

Related PR: apache#69886, apache#70075, apache#69343

Problem Summary: Master now provides paimon-rust v0.5.0-rc1 and Java Paimon
2.1.0. Rebase the writer on master instead of carrying separate dependency
upgrades. Reuse the upstream Java v14 serializer with bounded accumulated
output, remove the v11 and legacy DataFileMeta encoding patch, and regenerate
the real two-writer interop fixture. Reject old protocol requests before
writer admission. Preserve master's vector-search field 42 and use optional
field 43 for the shared descriptor; retain master's SPI additions and bump
the write diagnostics surface to API 30.

The previous BE UT crash was caused by a validation test bypassing init()
and reading an uninitialized scan-params pointer. Explicitly initialize its
missing-descriptor input.

### Release note

Rust Paimon writes use the Java v14 commit protocol matching master's Paimon
2.1.0 SDK and inherit master's paimon-rust dependency version.

### Check List (For Author)

- Test: Third-party build with Rust 1.94.0; real C FFI append, codec bounds,
  version rejection, message ranges and resource release; FE/Java build with
  Checkstyle; Paimon connector 904 tests (one skip); connector SPI 152 tests;
  six Paimon JNI static layout checks; Thrift generation/field audit;
  clang-format 16, header hygiene, shell syntax and whitespace checks.
  JNI Paimon 186 and FE integration 36 tests passed. BE compilation, native tests,
  clang-tidy against the new build and SQL regression are deferred at the
  user's request to leave resources for another PR.
- Behavior changed: Yes; Rust commit messages now use v14 instead of v11.
- Does this need documentation: Yes; existing feature documentation remains
  pending with the draft PR.
### What problem does this PR solve?

Related PR: apache#69886

Problem Summary: The upgraded paimon-rust writer needs native coverage of
credit release, cleanup and the v14 transport. Use the same Parquet buffer
option as the production descriptor instead of the generic write-buffer-size
option, which does not control the Parquet buffer.

Cover empty writes, prepare failure after rolling a file, and sustained
32 MiB dictionary input under a 16 MiB SDK input budget without rolling the
file. Extend the memory-failure case to retry the same batch with a larger
runtime limit. Check complete big-endian frame fields and the metadata
credit retained until report acknowledgement. The exercised paths required
no production logic changes.

### Release note

None

### Check List (For Author)

- Test: ASAN BE compilation and packaging with build.sh; 164 related BE
  tests passed through run-be-ut.sh (20 FFI writer tests, no failures or
  skips); clang-tidy for five modified production translation units and
  the full updated FFI test file; clang-format 16, header hygiene and
  whitespace checks. SQL/distributed regression was not rerun.
- Behavior changed: No; test-only changes.
- Does this need documentation: No; test-only changes.
@suxiaogang223
suxiaogang223 force-pushed the codex/branch-4.2-paimon-rust-write branch from 4078a16 to 2da24b4 Compare November 17, 2026 08:50
### Background

Paimon Rust writes send Java-compatible commit messages from BE to the FE committer.

### What problem does this PR solve?

Issue Number: apache#65086
Related PR: None
PR Type: Refactor

Problem Summary:

Use the released upstream writer interfaces and remove the local Rust patch. Keep Doris list framing around the individual Java v14 bodies returned by the range exporter.

### Details

- **Change map:** Replace the commit count and serialization calls, remove the third-party patch, and update asynchronous file lifecycle tests.
- **Moves vs. logic changes:** Adapt checked counts and range export; construct the existing list header in Doris.
- **Behavior change:** None for supported writes. Tests observe file completion after prepare or abort.
- **Compatibility:** Preserve the existing DPCM envelope and Java v14 list format.
- **Performance impact:** Export at most 512 messages and 8 MiB per payload; retain the 64 MiB total cap.
- **Rollback:** Revert this commit with its dependency baseline.

### Testing

Local ASAN BE and FE builds passed. FE/Java tests passed: 1,175 tests, one skipped. Append and two-BE failure regressions passed, including online memory adjustment, cleanup, and uncertain commit handling. All 57 selected BE tests passed, including 21 FFI writer tests. A Rust read after Rust writes returned matching results for 20,003 rows. clang-format 16 and header hygiene passed; clang-tidy passed for all 18 changed C++ files.

### Release note

None

### Check List (For Author)

- Behavior changed: No.
- Does this need documentation: No.
- Pre-submission status: Ready.

### AI Disclosure

- Tool: Codex (model ID unavailable in this session).
- Generated: Interface adaptation, tests, and this commit description.
- Human review: Current adaptation awaiting author review.
@suxiaogang223
suxiaogang223 force-pushed the codex/branch-4.2-paimon-rust-write branch from 2da24b4 to ce7f6e3 Compare November 17, 2026 09:53
@suxiaogang223

Copy link
Copy Markdown
Member Author

run buildall

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants