Skip to content

[fix](cloud-mow) Fix the issue of inaccurate estimation of txn size when updating delete bitmap - #46969

Merged
dataroaring merged 4 commits into
apache:masterfrom
hust-hhb:fix-txn-size
Jan 16, 2025
Merged

dataroaring merged 4 commits into
apache:masterfrom
hust-hhb:fix-txn-size

Conversation

@hust-hhb

@hust-hhb hust-hhb commented Jan 14, 2025 •

Copy link
Copy Markdown
Contributor

fdb_txn_size is not key size+value size, actually txn size is the triple of key size plus value size at least for writing, so use txn->approximate_bytes() to calculating txn size is more accurate. However, txn->approximate_bytes() is not 100% accurate either, in my test, when txn->approximate_bytes() bingger than 8.3MB, it will meet Transaction exceeds byte limit error.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas

Thearas commented Jan 14, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hust-hhb

Copy link
Copy Markdown
Contributor Author

run buildall

Comment thread cloud/src/common/config.h Resolved
Comment thread cloud/src/meta-service/meta_service.cpp Resolved
Comment thread cloud/src/meta-service/meta_service.cpp Resolved
@hust-hhb

Copy link
Copy Markdown
Contributor Author

run buildall

@zhannngchen zhannngchen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Jan 15, 2025
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@dataroaring dataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring
dataroaring merged commit 91d3a52 into apache:master Jan 16, 2025
@dataroaring dataroaring added the usercase Important user case type label label Jan 16, 2025
github-actions Bot pushed a commit that referenced this pull request Jan 16, 2025
…hen updating delete bitmap (#46969)

fdb_txn_size is not key size+value size, actually txn size is the triple
of key size plus value size at least for writing, so use
txn->approximate_bytes() to calculating txn size is more accurate.
However, txn->approximate_bytes() is not 100% accurate either, in my
test, when txn->approximate_bytes() bingger than 8.3MB, it will meet
Transaction exceeds byte limit error.
zhannngchen pushed a commit that referenced this pull request Jan 17, 2025
…f txn size when updating delete bitmap #46969 (#47046)

Cherry-picked from #46969

Co-authored-by: huanghaibin <huanghaibin@selectdb.com>
Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>
dataroaring pushed a commit that referenced this pull request Feb 11, 2025
This case is used to test the scenario of updating a large number of
delete bitmap when doing compaction, if delete bitmap size over the fdb
limit(10MB), updating delete bitmap will fail, pr #46969 fix this
problem, before the fix, this case will fail with error msg "Transaction
exceeds byte limit" on metaservice.
github-actions Bot pushed a commit that referenced this pull request Feb 11, 2025
This case is used to test the scenario of updating a large number of
delete bitmap when doing compaction, if delete bitmap size over the fdb
limit(10MB), updating delete bitmap will fail, pr #46969 fix this
problem, before the fix, this case will fail with error msg "Transaction
exceeds byte limit" on metaservice.
@gavinchou gavinchou mentioned this pull request Feb 18, 2025
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
…hen updating delete bitmap (apache#46969)

fdb_txn_size is not key size+value size, actually txn size is the triple
of key size plus value size at least for writing, so use
txn->approximate_bytes() to calculating txn size is more accurate.
However, txn->approximate_bytes() is not 100% accurate either, in my
test, when txn->approximate_bytes() bingger than 8.3MB, it will meet
Transaction exceeds byte limit error.
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
This case is used to test the scenario of updating a large number of
delete bitmap when doing compaction, if delete bitmap size over the fdb
limit(10MB), updating delete bitmap will fail, pr apache#46969 fix this
problem, before the fix, this case will fail with error msg "Transaction
exceeds byte limit" on metaservice.
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
This case is used to test the scenario of updating a large number of
delete bitmap when doing compaction, if delete bitmap size over the fdb
limit(10MB), updating delete bitmap will fail, pr apache#46969 fix this
problem, before the fix, this case will fail with error msg "Transaction
exceeds byte limit" on metaservice.
suxiaogang223 pushed a commit to suxiaogang223/doris that referenced this pull request Jul 15, 2025
bobhan1 added a commit to bobhan1/doris that referenced this pull request Nov 11, 2026
### Background

Cloud Merge-on-Write stores each logical delete bitmap as a blob of 90 KB KV fragments. It already batches different bitmaps into separate metadata transactions before publishing the load version or output rowset.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#46969 (transaction budgeting), apache#47097 (large-bitmap tests)
PR Type: Bugfix

Problem Summary:

A single bitmap exceeding the FDB transaction limit still fails because all of its KV fragments are written in one transaction. Apply the existing transaction budget between fragments of oversized locked bitmap writes.

### Details

- **Change map:** 44 changed lines in one production function, 362 lines of tests, and one test-target registration.
- **Frequency:** A single large v1 bitmap or serialized v2 storage value, such as a 10,578,008-byte bitmap.
- **Root cause:** The budget check runs between logical values; `blob_put` splits values into KVs within one transaction.
- **Why existing tests missed it:** MemTxnKv does not enforce FDB transaction-size limits.
- **Fix:** Reuse the existing commit-and-renew path between 90 KB fragments when a locked value exceeds `max_txn_commit_byte`. Keep the existing lock checks, pending cleanup, fragment encoding, and configuration.
- **Other paths and branches:** Cover v1/v2/dual writes, load, compaction, and schema change on master. Unlocked aggregation retains its atomic overwrite behavior because it updates visible bitmaps.
- **Behavior change:** One oversized locked bitmap can span metadata transactions while retaining the existing Doris publish boundary.
- **Compatibility:** No protocol or persistent-format changes.

### Testing

- Before the fix, v1/v2 large writes, ambiguous-commit retry, and lock-loss recovery cases failed with transaction-size errors.
- ASAN: `BUILD_TYPE=ASAN ./run-cloud-ut.sh --run --filter='meta_service_test:*DeleteBitmap*' -j100`: 36 tests passed, including nine new cases. The memory store enforces transaction/KV size limits; tests verify readback, retry cleanup, explicit sub-transactions, and atomic unlocked overwrites.
- Production Cloud ASAN build passed via `./build.sh --cloud -j100`, with tests and injection points disabled.
- Format and `git diff --check` passed. clang-tidy was skipped at the author's request. No real-FDB or FE/BE end-to-end test was run.

### Release note

Fix Cloud Merge-on-Write writes failing when a single delete bitmap exceeds the FDB transaction limit.

### Check List (For Author)

- Behavior changed: Yes (see Details).
- Does this need documentation: No.
- Pre-submission status: Partially verified: unit tests and production compilation passed; real-FDB and cluster validation remain.

### AI Disclosure

- Tool: Codex (GPT-6).
- Generated: Implementation, tests, and this description.
- Human review: Pending author/maintainer review.
bobhan1 added a commit to bobhan1/doris that referenced this pull request Nov 12, 2026
### Background

Cloud Merge-on-Write stores each logical delete bitmap as a blob of 90 KB KV fragments. It already batches different bitmaps into separate metadata transactions before publishing the load version or output rowset.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#46969 (transaction budgeting), apache#47097 (large-bitmap tests)
PR Type: Bugfix

Problem Summary:

A single bitmap exceeding the FDB transaction limit still fails because all of its KV fragments are written in one transaction. Apply the existing transaction budget between fragments of oversized locked bitmap writes.

### Details

- **Change map:** 44 changed lines in one production function, 417 lines of tests, and one test-target registration.
- **Frequency:** A single large v1 bitmap or serialized v2 storage value, such as a 10,578,008-byte bitmap.
- **Root cause:** The budget check runs between logical values; `blob_put` splits values into KVs within one transaction.
- **Why existing tests missed it:** MemTxnKv does not enforce FDB transaction-size limits.
- **Fix:** Reuse the existing commit-and-renew path between 90 KB fragments when a locked value exceeds `max_txn_commit_byte`. Keep the existing lock checks, pending cleanup, fragment encoding, and configuration.
- **Other paths and branches:** Cover v1/v2/dual writes, load, compaction, and schema change on master. Unlocked aggregation retains its atomic overwrite behavior because it updates visible bitmaps.
- **Behavior change:** One oversized locked bitmap can span metadata transactions while retaining the existing Doris publish boundary.
- **Compatibility:** No protocol or persistent-format changes.

### Testing

- Before the fix, v1/v2 large writes, ambiguous-commit retry, and lock-loss recovery cases failed with transaction-size errors.
- ASAN: the related bitmap suite passed 36 tests. After adding the backend selector, the same nine new cases passed against both a private **FoundationDB 7.3.69** instance and MemTxnKv.
- FDB command: `DORIS_LARGE_BITMAP_FDB_CLUSTER_FILE=/path/to/fdb.cluster BUILD_TYPE=ASAN ./run-cloud-ut.sh --run --filter='meta_service_test:LargeDeleteBitmapTest.*' --fdb='<connection-string>' -j100`. Configure the bundled 7.3.69 external-client directory for a 7.3 server; the default client API is 710.
- FDB coverage: 10,578,008-byte values, v1/v2/dual readback, compaction/schema change, explicit sub-transactions, lock loss and shorter retry/tail cleanup. An ambiguous result is injected after a real commit. The atomic unlocked-overwrite case receives the actual FDB 2101 size error. GTest XML records the backend for every case; test keys in both metadata key spaces were cleaned up.
- Production Cloud ASAN compilation previously passed with tests and injection points disabled; production sources are unchanged by the FDB test addition.
- Format and `git diff --check` passed. clang-tidy was skipped at the author's request. No FE/BE end-to-end test was run.

### Release note

Fix Cloud Merge-on-Write writes failing when a single delete bitmap exceeds the FDB transaction limit.

### Check List (For Author)

- Behavior changed: Yes (see Details).
- Does this need documentation: No.
- Pre-submission status: Partially verified: unit tests against MemTxnKv and FoundationDB passed, with production compilation; no FE/BE end-to-end run.

### AI Disclosure

- Tool: Codex (GPT-6).
- Generated: Implementation, tests, and this description.
- Human review: Pending author/maintainer review.
bobhan1 added a commit to bobhan1/doris that referenced this pull request Nov 12, 2026
### Background

Cloud Merge-on-Write stores each logical delete bitmap as a blob of 90 KB KV fragments. It already batches different bitmaps into separate metadata transactions before publishing the load version or output rowset.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#46969 (transaction budgeting), apache#47097 (large-bitmap tests)
PR Type: Bugfix

Problem Summary:

A single bitmap exceeding the FDB transaction limit still fails because all of its KV fragments are written in one transaction. Apply the existing transaction budget between fragments of oversized locked bitmap writes.

### Details

- **Change map:** 44 changed lines in one production function, 423 lines of tests, two runner lines, and one test-target registration.
- **Frequency:** A single large v1 bitmap or serialized v2 storage value, such as a 10,578,008-byte bitmap.
- **Root cause:** The budget check runs between logical values; `blob_put` splits values into KVs within one transaction.
- **Why existing tests missed it:** MemTxnKv does not enforce FDB transaction-size limits.
- **Fix:** Reuse the existing commit-and-renew path between 90 KB fragments when a locked value exceeds `max_txn_commit_byte`. Keep the existing lock checks, pending cleanup, fragment encoding, and configuration.
- **Other paths and branches:** Cover v1/v2/dual writes, load, compaction, and schema change on master. Unlocked aggregation retains its atomic overwrite behavior because it updates visible bitmaps.
- **Behavior change:** One oversized locked bitmap can span metadata transactions while retaining the existing Doris publish boundary.
- **Compatibility:** No protocol or persistent-format changes.

### Testing

- Before the fix, v1/v2 large writes, ambiguous-commit retry, and lock-loss recovery cases failed with transaction-size errors. The related ASAN bitmap suite previously passed 36 tests.
- The focused suite now has 11 cases, including 32 MiB single-value readback for both v1 and v2. **All 11 passed against FoundationDB 7.3.69, and all 11 passed against MemTxnKv.** The 32 MiB FDB cases each logged six commits: one for pending metadata and five for data.
- `BUILD_TYPE=ASAN ./run-cloud-ut.sh --run --filter='meta_service_test:LargeDeleteBitmapTest.*' --fdb='<connection-string>' -j100` selects FDB through the existing runner flag. Omitting `--fdb` selects MemTxnKv for this suite. No extra user environment setting is required. A 7.3 server uses the bundled 7.3.69 external client with the default API 710 client.
- Coverage includes 10,578,008-byte and 32 MiB values, v1/v2/dual readback, compaction/schema change, explicit sub-transactions, lock loss and shorter retry/tail cleanup. An ambiguous result is injected after a real commit. The atomic unlocked-overwrite case receives the actual FDB 2101 size error. XML records each case's backend and the larger cases' input size; both metadata key spaces were cleaned up.
- Production Cloud ASAN compilation previously passed with tests and injection points disabled; production sources are unchanged by these test/runner additions.
- Format, shell syntax, and `git diff --check` passed. clang-tidy was skipped at the author's request. No FE/BE end-to-end test was run.

### Release note

Fix Cloud Merge-on-Write writes failing when a single delete bitmap exceeds the FDB transaction limit.

### Check List (For Author)

- Behavior changed: Yes (see Details).
- Does this need documentation: No.
- Pre-submission status: Partially verified: unit tests against MemTxnKv and FoundationDB passed, with production compilation; no FE/BE end-to-end run.

### AI Disclosure

- Tool: Codex (GPT-6).
- Generated: Implementation, tests, and this description.
- Human review: Pending author/maintainer review.
bobhan1 added a commit to bobhan1/doris that referenced this pull request Nov 12, 2026
### Background

Cloud Merge-on-Write stores each logical delete bitmap as a blob of 90 KB KV fragments. It already batches different bitmaps into separate metadata transactions before publishing the load version or output rowset.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#46969 (transaction budgeting), apache#47097 (large-bitmap tests)
PR Type: Bugfix

Problem Summary:

A single bitmap exceeding the FDB transaction limit still fails because all of its KV fragments are written in one transaction. Apply the existing transaction budget between fragments of oversized locked bitmap writes.

### Details

- **Change map:** 44 changed lines in one production function, 423 lines of tests, and one test-target registration.
- **Frequency:** A single large v1 bitmap or serialized v2 storage value, such as a 10,578,008-byte bitmap.
- **Root cause:** The budget check runs between logical values; `blob_put` splits values into KVs within one transaction.
- **Why existing tests missed it:** MemTxnKv does not enforce FDB transaction-size limits.
- **Fix:** Reuse the existing commit-and-renew path between 90 KB fragments when a locked value exceeds `max_txn_commit_byte`. Keep the existing lock checks, pending cleanup, fragment encoding, and configuration.
- **Other paths and branches:** Cover v1/v2/dual writes, load, compaction, and schema change on master. Unlocked aggregation retains its atomic overwrite behavior because it updates visible bitmaps.
- **Behavior change:** One oversized locked bitmap can span metadata transactions while retaining the existing Doris publish boundary.
- **Compatibility:** No protocol or persistent-format changes.

### Testing

- Before the fix, v1/v2 large writes, ambiguous-commit retry, and lock-loss recovery cases failed with transaction-size errors. The related ASAN bitmap suite previously passed 36 tests.
- Eleven parameterized cases are instantiated for both MemTxnKv and FdbTxnKv. **All 22 passed in one ASAN run**, including 32 MiB single-value readback for both v1 and v2 on both backends. The two larger FDB cases each logged six commits: one for pending metadata and five for data.
- Command: `BUILD_TYPE=ASAN ./run-cloud-ut.sh --run --filter='meta_service_test:*LargeDeleteBitmapTest*' --fdb='<connection-string>' -j100`. Follow existing FDB tests: set `fdb_cluster_file_path` to `fdb.cluster` and construct FdbTxnKv directly. The existing `--fdb` runner option only supplies the connection file; no runner changes or extra environment variables are needed. Filters `Mem/LargeDeleteBitmapTest.*` and `Fdb/LargeDeleteBitmapTest.*` select individual backends.
- Coverage includes 10,578,008-byte and 32 MiB values, v1/v2/dual readback, compaction/schema change, explicit sub-transactions, lock loss and shorter retry/tail cleanup. An ambiguous result is injected after a real commit. The atomic unlocked-overwrite case receives the actual FDB 2101 size error. XML records backend and input size; both metadata key spaces were cleaned up. FDB 7.3.69 was tested with the bundled external client and default API 710 client.
- Production Cloud ASAN compilation previously passed with tests and injection points disabled; production sources are unchanged by these test revisions.
- Format and `git diff --check` passed. clang-tidy was skipped at the author's request. No FE/BE end-to-end test was run.

### Release note

Fix Cloud Merge-on-Write writes failing when a single delete bitmap exceeds the FDB transaction limit.

### Check List (For Author)

- Behavior changed: Yes (see Details).
- Does this need documentation: No.
- Pre-submission status: Partially verified: unit tests against MemTxnKv and FoundationDB passed, with production compilation; no FE/BE end-to-end run.

### AI Disclosure

- Tool: Codex (GPT-6).
- Generated: Implementation, tests, and this description.
- Human review: Pending author/maintainer review.
bobhan1 added a commit to bobhan1/doris that referenced this pull request Nov 12, 2026
### Background

Cloud Merge-on-Write stores each logical delete bitmap as a blob of 90 KB KV fragments. It already batches different bitmaps into separate metadata transactions before publishing the load version or output rowset.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#46969 (transaction budgeting), apache#47097 (large-bitmap tests)
PR Type: Bugfix

Problem Summary:

A single bitmap exceeding the FDB transaction limit still fails because all of its KV fragments are written in one transaction. Apply the existing transaction budget between fragments of oversized locked bitmap writes. Extract transaction renewal, rowset checks, and pre-update removal into helpers while retaining the existing cumulative budget check between values.

### Details

- **Change map:** 376 changed production lines, mostly extracting three existing blocks into helpers, 470 lines of tests, and one test-target registration. The bitmap writer shrinks from 211 to 76 lines.
- **Frequency:** A single large v1 bitmap or serialized v2 storage value, such as a 10,578,008-byte bitmap.
- **Root cause:** The budget check runs between logical values; `blob_put` splits values into KVs within one transaction.
- **Why existing tests missed it:** MemTxnKv does not enforce FDB transaction-size limits.
- **Fix:** Reuse the existing commit-and-renew path between 90 KB fragments when a locked value exceeds `max_txn_commit_byte`. Preserve lock checks, pending cleanup, fragment encoding, and configuration. Separate commit/renewal, rowset validation, and unlocked overwrite cleanup from the main writer; preserve the distinction between skipping one bitmap and stopping an obsolete aggregation request.
- **Other paths and branches:** Cover v1/v2/dual writes, load, compaction, and schema change on master. Unlocked aggregation retains its atomic overwrite behavior because it updates visible bitmaps.
- **Behavior change:** One oversized locked bitmap can span metadata transactions while retaining the existing Doris publish boundary.
- **Compatibility:** No protocol or persistent-format changes.

### Testing

- Before the fix, v1/v2 large writes, ambiguous-commit retry, and lock-loss recovery cases failed with transaction-size errors.
- After extraction, ASAN bitmap tests passed **53/53**, and pre-rowset tests passed **3/3** (55 unique cases). This includes 13 parameterized cases on each of MemTxnKv and real FoundationDB 7.3.69, plus existing lock, point/range deletion, obsolete aggregation, malformed metadata, and cleanup tests.
- The two-value cases send two 6 MiB values in one request: each is below the 7 MiB budget, while their sum exceeds FDB's hard limit. Both v1 and v2 read back correctly on both backends; FDB logs show two commits. Single 32 MiB v1/v2 values each use six commits, including pending metadata, and read back completely.
- Commands: `BUILD_TYPE=ASAN ./run-cloud-ut.sh --run --filter='meta_service_test:*DeleteBitmap*' --fdb='<connection-string>' -j100` and the same command with `*PreRowset*`. FDB cases directly read `fdb.cluster`; the existing `--fdb` option only supplies that file. No extra environment variables or runner changes are needed.
- Ambiguous results are injected after a real commit. Coverage includes dual writes, compaction/schema change, explicit sub-transactions, lock loss and shorter retry/tail cleanup, and an actual FDB 2101 size error for atomic unlocked overwrites. Both metadata key spaces were cleaned up. FDB 7.3.69 uses the bundled external client with the default API 710 client.
- Production Cloud ASAN build passed via `./build.sh --cloud -j100`, with `MAKE_TEST=OFF`, injection points disabled, and Java extensions disabled.
- Format and `git diff --check` passed. clang-tidy was skipped at the author's request. No FE/BE end-to-end test was run.

### Release note

Fix Cloud Merge-on-Write writes failing when a single delete bitmap exceeds the FDB transaction limit.

### Check List (For Author)

- Behavior changed: Yes (see Details).
- Does this need documentation: No.
- Pre-submission status: Partially verified: unit tests against MemTxnKv and FoundationDB passed, with production compilation; no FE/BE end-to-end run.

### AI Disclosure

- Tool: Codex (GPT-6).
- Generated: Implementation, tests, and this description.
- Human review: Pending author/maintainer review.
liaoxin01 pushed a commit that referenced this pull request Nov 16, 2026
…ons (#70478)

### Background

Cloud Merge-on-Write stores each logical delete bitmap as a blob of 90
KB KV fragments. It already batches different bitmaps into separate
metadata transactions before publishing the load version or output
rowset.

### What problem does this PR solve?

Issue Number: None
Related PR: #46969 (transaction budgeting), #47097 (large-bitmap tests)
PR Type: Bugfix

Problem Summary:

A single bitmap exceeding the FDB transaction limit still fails because
all of its KV fragments are written in one transaction. Apply the
existing transaction budget between fragments of oversized locked bitmap
writes, and verify the committed prefix in the last data transaction so
overlapping RPC cleanup cannot leave a successful suffix-only write.
Extract transaction renewal, rowset checks, and pre-update removal into
helpers while retaining the existing cumulative budget check between
values.

### Details

- **Change map:** 411 changed production lines, mostly extracting three
existing blocks into helpers, 544 lines of tests, and one test-target
registration. The bitmap writer shrinks from 211 to 85 lines; a separate
29-line helper verifies the committed prefix.
- **Frequency:** A single large v1 bitmap or serialized v2 storage
value, such as a 10,578,008-byte bitmap.
- **Root cause:** The budget check runs between logical values;
`blob_put` splits values into KVs within one transaction.
- **Why existing tests missed it:** MemTxnKv does not enforce FDB
transaction-size limits. Sequential retries also miss pending cleanup
interleaved with fragment writes.
- **Fix:** Reuse the existing commit-and-renew path between 90 KB
fragments when a locked value exceeds `max_txn_commit_byte`. In the last
data transaction, read the committed prefix without snapshot reads and
verify every fragment's key and contents. A missing or changed prefix
returns `KV_TXN_CONFLICT`; cleanup after that read conflicts with the
commit. Preserve lock checks, pending cleanup, fragment encoding, and
configuration. Separate commit/renewal, rowset validation, and unlocked
overwrite cleanup from the main writer.
- **Other paths and branches:** Cover v1/v2/dual writes, load,
compaction, and schema change on master. Unlocked aggregation retains
its atomic overwrite behavior because it updates visible bitmaps.
- **Behavior change:** One oversized locked bitmap can span metadata
transactions while retaining the existing Doris publish boundary.
- **Compatibility:** No protocol or persistent-format changes.
Fragmented values require one additional read of their committed prefix;
other values retain their existing write path.

### Testing

- Before the fix, v1/v2 large writes, ambiguous-commit retry, and
lock-loss recovery cases failed with transaction-size errors.
- ASAN bitmap tests passed **58**, with **1 skipped**; pre-rowset tests
passed **3/3**. After deduplication, **60 passed and 1 skipped**.
FoundationDB 7.3.69 passed all 16 parameterized cases; MemTxnKv passed
15 and skipped the post-validation cleanup conflict case because it does
not detect read conflicts with covering range clears. Existing lock,
point/range deletion, obsolete aggregation, malformed metadata, and
cleanup cases also passed.
- Before prefix validation, all four new cleanup/replacement test cases
failed on MemTxnKv and real FDB, returning OK after prefix loss or
replacement. The corrected tests cover 8 MiB and 32 MiB values in both
v1 and v2, pending cleanup before validation, cleanup after validation
but before commit, changed prefix contents, and successful full readback
after retry.
- The two-value cases send two 6 MiB values in one request: each is
below the 7 MiB budget, while their sum exceeds FDB's hard limit. Both
v1 and v2 read back correctly on both backends; FDB logs show two
commits. Single 32 MiB v1/v2 values each use six commits, including
pending metadata, and read back completely.
- Commands: `BUILD_TYPE=ASAN ./run-cloud-ut.sh --run
--filter='meta_service_test:*DeleteBitmap*' --fdb='<connection-string>'
-j100` and the same command with `*PreRowset*`. FDB cases directly read
`fdb.cluster`; the existing `--fdb` option only supplies that file. No
extra environment variables or runner changes are needed.
- Ambiguous results are injected after a real commit. Coverage includes
dual writes, compaction/schema change, explicit sub-transactions, lock
loss and shorter retry/tail cleanup, and an actual FDB 2101 size error
for atomic unlocked overwrites. Both metadata key spaces were cleaned
up. FDB 7.3.69 uses the bundled external client with the default API 710
client.
- Production Cloud ASAN build passed via `./build.sh --cloud -j100`,
with `MAKE_TEST=OFF`, injection points disabled, and Java extensions
disabled.
- Format and `git diff --check` passed. clang-tidy was skipped at the
author's request. No FE/BE end-to-end test was run.

### Notes for Reviewers

Prefix validation restores bitmap completeness at the last data commit.
The pre-existing race involving a late cleanup after an RPC has already
returned OK is outside this change.

### Release note

Fix Cloud Merge-on-Write writes failing when a single delete bitmap
exceeds the FDB transaction limit.
github-actions Bot pushed a commit that referenced this pull request Nov 16, 2026
…ons (#70478)

### Background

Cloud Merge-on-Write stores each logical delete bitmap as a blob of 90
KB KV fragments. It already batches different bitmaps into separate
metadata transactions before publishing the load version or output
rowset.

### What problem does this PR solve?

Issue Number: None
Related PR: #46969 (transaction budgeting), #47097 (large-bitmap tests)
PR Type: Bugfix

Problem Summary:

A single bitmap exceeding the FDB transaction limit still fails because
all of its KV fragments are written in one transaction. Apply the
existing transaction budget between fragments of oversized locked bitmap
writes, and verify the committed prefix in the last data transaction so
overlapping RPC cleanup cannot leave a successful suffix-only write.
Extract transaction renewal, rowset checks, and pre-update removal into
helpers while retaining the existing cumulative budget check between
values.

### Details

- **Change map:** 411 changed production lines, mostly extracting three
existing blocks into helpers, 544 lines of tests, and one test-target
registration. The bitmap writer shrinks from 211 to 85 lines; a separate
29-line helper verifies the committed prefix.
- **Frequency:** A single large v1 bitmap or serialized v2 storage
value, such as a 10,578,008-byte bitmap.
- **Root cause:** The budget check runs between logical values;
`blob_put` splits values into KVs within one transaction.
- **Why existing tests missed it:** MemTxnKv does not enforce FDB
transaction-size limits. Sequential retries also miss pending cleanup
interleaved with fragment writes.
- **Fix:** Reuse the existing commit-and-renew path between 90 KB
fragments when a locked value exceeds `max_txn_commit_byte`. In the last
data transaction, read the committed prefix without snapshot reads and
verify every fragment's key and contents. A missing or changed prefix
returns `KV_TXN_CONFLICT`; cleanup after that read conflicts with the
commit. Preserve lock checks, pending cleanup, fragment encoding, and
configuration. Separate commit/renewal, rowset validation, and unlocked
overwrite cleanup from the main writer.
- **Other paths and branches:** Cover v1/v2/dual writes, load,
compaction, and schema change on master. Unlocked aggregation retains
its atomic overwrite behavior because it updates visible bitmaps.
- **Behavior change:** One oversized locked bitmap can span metadata
transactions while retaining the existing Doris publish boundary.
- **Compatibility:** No protocol or persistent-format changes.
Fragmented values require one additional read of their committed prefix;
other values retain their existing write path.

### Testing

- Before the fix, v1/v2 large writes, ambiguous-commit retry, and
lock-loss recovery cases failed with transaction-size errors.
- ASAN bitmap tests passed **58**, with **1 skipped**; pre-rowset tests
passed **3/3**. After deduplication, **60 passed and 1 skipped**.
FoundationDB 7.3.69 passed all 16 parameterized cases; MemTxnKv passed
15 and skipped the post-validation cleanup conflict case because it does
not detect read conflicts with covering range clears. Existing lock,
point/range deletion, obsolete aggregation, malformed metadata, and
cleanup cases also passed.
- Before prefix validation, all four new cleanup/replacement test cases
failed on MemTxnKv and real FDB, returning OK after prefix loss or
replacement. The corrected tests cover 8 MiB and 32 MiB values in both
v1 and v2, pending cleanup before validation, cleanup after validation
but before commit, changed prefix contents, and successful full readback
after retry.
- The two-value cases send two 6 MiB values in one request: each is
below the 7 MiB budget, while their sum exceeds FDB's hard limit. Both
v1 and v2 read back correctly on both backends; FDB logs show two
commits. Single 32 MiB v1/v2 values each use six commits, including
pending metadata, and read back completely.
- Commands: `BUILD_TYPE=ASAN ./run-cloud-ut.sh --run
--filter='meta_service_test:*DeleteBitmap*' --fdb='<connection-string>'
-j100` and the same command with `*PreRowset*`. FDB cases directly read
`fdb.cluster`; the existing `--fdb` option only supplies that file. No
extra environment variables or runner changes are needed.
- Ambiguous results are injected after a real commit. Coverage includes
dual writes, compaction/schema change, explicit sub-transactions, lock
loss and shorter retry/tail cleanup, and an actual FDB 2101 size error
for atomic unlocked overwrites. Both metadata key spaces were cleaned
up. FDB 7.3.69 uses the bundled external client with the default API 710
client.
- Production Cloud ASAN build passed via `./build.sh --cloud -j100`,
with `MAKE_TEST=OFF`, injection points disabled, and Java extensions
disabled.
- Format and `git diff --check` passed. clang-tidy was skipped at the
author's request. No FE/BE end-to-end test was run.

### Notes for Reviewers

Prefix validation restores bitmap completeness at the last data commit.
The pre-existing race involving a late cleanup after an RPC has already
returned OK is outside this change.

### Release note

Fix Cloud Merge-on-Write writes failing when a single delete bitmap
exceeds the FDB transaction limit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.4-merged reviewed usercase Important user case type label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants