Skip to content

branch-4.1: [fix](iceberg) Commit every Iceberg file under the table's own location #69392 - #70172

Open
xylaaaaa wants to merge 3 commits into
apache:branch-4.1from
xylaaaaa:codex/pick-69392-4.1
Open

xylaaaaa wants to merge 3 commits into
apache:branch-4.1from
xylaaaaa:codex/pick-69392-4.1

Conversation

@xylaaaaa

@xylaaaaa xylaaaaa commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: #69392

Problem Summary:
Backport #69392 to branch-4.1. Iceberg DELETE/UPDATE/MERGE could record position-delete and Puffin files under BE's normalized s3:// address even when the table uses oss://, s3a://, or abfss://. These metadata paths can prevent Doris and other engines from reading the files.

BE now derives physical I/O paths and raw metadata paths from one output location for data files, position deletes, and deletion vectors. FE registers the raw locations from the actual bound sinks and validates the entire collected file list before staging INSERT/overwrite, DELETE, MERGE, or rewrite updates.

Release-branch adaptation:

  • Use the existing typed Iceberg transaction and planner sinks. Validate at transaction completion rather than in the report RPC, since these branches do not have [fix](iceberg) Harden external-write lifecycle, OCC, memory, and file ownership #66348's report ACK mechanism.
  • Register rewrite group locations after binding the shared transaction; preserve empty static partition overwrites, which execute without a BE sink.
  • Preserve the branch's existing failed-file cleanup behavior. This backport does not add ACK-based ownership transfer or rollback file deletion.
  • No wire-format changes; the Thrift changes are comments only.

Release note

Fix Iceberg files being committed with normalized storage addresses instead of their raw table data locations. Reject out-of-location file paths before staging Iceberg updates. Upgrade BE before FE: an older BE that reports normalized delete-file paths will be rejected by the upgraded FE. Failed writes can still leave uncommitted files under the release branch's existing cleanup behavior.

Check List (For Author)

  • Test:
    • Unit Test: 85 focused FE tests passed via run-fe-ut.sh --run, covering transactions, all four completion paths, raw/normalized and directory-prefix rejection, sink registration, empty static overwrite, rewrite groups, and SQL failure propagation.
    • FE packaging and ASAN BE build passed via build.sh; the generated BE binary passed --version. FE UI was skipped. Local Clang 22 compatibility options were used for baseline diagnostics and are not part of this PR.
    • Java Checkstyle passed for the final changed files; C++ clang-format 16 checks passed.
    • BE unit tests: local execution is not verified; validate the complete suite in CI.
    • Iceberg regression: cases are included; not run locally because the external fixture/Docker daemon is unavailable.
    • C++ static analysis was attempted; the full run is not green due to existing macro/NOLINT/complexity diagnostics. No broad suppressions or environment changes are included.
  • Behavior changed:
    • Yes: metadata paths use raw locations, and invalid reported paths fail the statement.
  • Does this need documentation?
    • No.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

xylaaaaa and others added 2 commits November 4, 2026 10:59
### What problem does this PR solve?

Issue Number: None

Related PR: apache#69392

Problem Summary: Iceberg delete and Puffin files were committed using the
normalized storage URI instead of the table's original data location. Other
engines could resolve those paths with the wrong file system. Carry the write
and metadata locations together for data, position-delete, and Puffin files,
join trailing separators consistently, and retain the write path for cleanup.

Backport the BE path fix and relevant tests to branch-4.1. Adapt writer
tests to the release branch interfaces and add planner coverage for raw versus
normalized paths. Omit FE commit-location rejection and the external-file
report ACK prerequisite from apache#66348, as those mechanisms are not on this branch.

### Release note

Preserve the table's original URI in newly written Iceberg file metadata,
including position deletes and deletion vectors, and handle trailing slashes
in custom data locations consistently.

### Check List (For Author)

- Test: Unit Test - 16 FE planner sink tests passed; clang-format 16, Java
  Checkstyle, patch comparison, and whitespace checks passed. BE UT build was
  attempted but blocked by macOS/Clang baseline incompatibilities. clang-tidy
  reported macro/baseline lint diagnostics. Iceberg regression was not run
  because the external fixture was unavailable.
- Behavior changed: Yes - newly committed file paths retain the table's URI;
  sinks missing required locations fail before creating files.
- Does this need documentation: No

Adapted partial backport of commit b78c656.
### What problem does this PR solve?

Issue Number: None

Related PR: apache#69392

Problem Summary: Release branches accept Iceberg file reports without validating
their raw sink locations. A writer reporting a normalized s3:// path for an
oss://, s3a:// or abfss:// location can publish an unreadable file address.
Register the locations from the bound sinks and validate the entire collected
file list before staging INSERT, overwrite, DELETE, MERGE or rewrite updates.
Propagate failure through the existing transaction completion path. Preserve
empty static overwrites and register rewrite groups on their shared transaction.
This adapts the FE guard without the external-report ACK and cleanup changes.

### Release note

Reject Iceberg writes whose reported files are outside their sink's raw
locations. Upgrade BE before FE when applying the location fix. Failed-file
cleanup retains the release branch's existing behavior.

### Check List (For Author)

- Test: 82 focused FE unit tests passed via run-fe-ut.sh; Maven Checkstyle passed.
    - Unit Test: transaction staging, path rejection, executor registration,
      empty static overwrite, SQL error propagation and existing sink tests.
- Behavior changed: Yes; invalid file paths fail before Iceberg updates are staged.
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@xylaaaaa
xylaaaaa marked this pull request as ready for review November 4, 2026 06:22
@xylaaaaa
xylaaaaa requested a review from yiguolei as a code owner November 4, 2026 06:22
@xylaaaaa

xylaaaaa commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

Copilot AI balanced review requested due to automatic review settings November 4, 2026 06:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@xylaaaaa xylaaaaa changed the title [fix](iceberg) Commit files under table locations (branch-4.1) branch-4.1: [fix](iceberg) Commit every Iceberg file under the table's own location #69392 Nov 4, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#69392

Problem Summary: The backported file-location regression was missing from the
Iceberg P0 coverage matrix. Add risk R19, case W20, and the suite mapping from
upstream so the matrix includes all supported write operations.

### Release note

None

### Check List (For Author)

- Test: Manual test; verified the coverage mapping against the backported regression and upstream PR. git diff --check passed.
- Behavior changed: No
- Does this need documentation: Yes; update the requested existing coverage matrix in this commit.
@xylaaaaa

xylaaaaa commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 91.43% (32/35) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 95.59% (65/68) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.28% (32346/42968)
Line Coverage 59.72% (363013/607867)
Region Coverage 56.37% (302468/536562)
Branch Coverage 57.35% (137802/240268)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants