Repository navigation
[fix](cloud-mow) Fix the issue of inaccurate estimation of txn size when updating delete bitmap - #46969
Merged
Conversation
…hen updating delete bitmap
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
zhannngchen
reviewed
Jan 14, 2025
zhannngchen
reviewed
Jan 14, 2025
zhannngchen
reviewed
Jan 14, 2025
Contributor
Author
|
run buildall |
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
w41ter
approved these changes
Jan 15, 2025
github-actions Bot
pushed a commit
that referenced
this pull request
Jan 16, 2025
…hen updating delete bitmap (#46969) fdb_txn_size is not key size+value size, actually txn size is the triple of key size plus value size at least for writing, so use txn->approximate_bytes() to calculating txn size is more accurate. However, txn->approximate_bytes() is not 100% accurate either, in my test, when txn->approximate_bytes() bingger than 8.3MB, it will meet Transaction exceeds byte limit error.
16 tasks
dataroaring
pushed a commit
that referenced
this pull request
Feb 11, 2025
This case is used to test the scenario of updating a large number of delete bitmap when doing compaction, if delete bitmap size over the fdb limit(10MB), updating delete bitmap will fail, pr #46969 fix this problem, before the fix, this case will fail with error msg "Transaction exceeds byte limit" on metaservice.
github-actions Bot
pushed a commit
that referenced
this pull request
Feb 11, 2025
This case is used to test the scenario of updating a large number of delete bitmap when doing compaction, if delete bitmap size over the fdb limit(10MB), updating delete bitmap will fail, pr #46969 fix this problem, before the fix, this case will fail with error msg "Transaction exceeds byte limit" on metaservice.
Closed
lzyy2024
pushed a commit
to lzyy2024/doris
that referenced
this pull request
Feb 21, 2025
…hen updating delete bitmap (apache#46969) fdb_txn_size is not key size+value size, actually txn size is the triple of key size plus value size at least for writing, so use txn->approximate_bytes() to calculating txn size is more accurate. However, txn->approximate_bytes() is not 100% accurate either, in my test, when txn->approximate_bytes() bingger than 8.3MB, it will meet Transaction exceeds byte limit error.
lzyy2024
pushed a commit
to lzyy2024/doris
that referenced
this pull request
Feb 21, 2025
This case is used to test the scenario of updating a large number of delete bitmap when doing compaction, if delete bitmap size over the fdb limit(10MB), updating delete bitmap will fail, pr apache#46969 fix this problem, before the fix, this case will fail with error msg "Transaction exceeds byte limit" on metaservice.
koarz
pushed a commit
to koarz/doris
that referenced
this pull request
Jun 4, 2025
This case is used to test the scenario of updating a large number of delete bitmap when doing compaction, if delete bitmap size over the fdb limit(10MB), updating delete bitmap will fail, pr apache#46969 fix this problem, before the fix, this case will fail with error msg "Transaction exceeds byte limit" on metaservice.
suxiaogang223
pushed a commit
to suxiaogang223/doris
that referenced
this pull request
Jul 15, 2025
…f txn size when updating delete bitmap apache#46969 (apache#3920)
bobhan1
added a commit
to bobhan1/doris
that referenced
this pull request
Nov 11, 2026
### Background Cloud Merge-on-Write stores each logical delete bitmap as a blob of 90 KB KV fragments. It already batches different bitmaps into separate metadata transactions before publishing the load version or output rowset. ### What problem does this PR solve? Issue Number: None Related PR: apache#46969 (transaction budgeting), apache#47097 (large-bitmap tests) PR Type: Bugfix Problem Summary: A single bitmap exceeding the FDB transaction limit still fails because all of its KV fragments are written in one transaction. Apply the existing transaction budget between fragments of oversized locked bitmap writes. ### Details - **Change map:** 44 changed lines in one production function, 362 lines of tests, and one test-target registration. - **Frequency:** A single large v1 bitmap or serialized v2 storage value, such as a 10,578,008-byte bitmap. - **Root cause:** The budget check runs between logical values; `blob_put` splits values into KVs within one transaction. - **Why existing tests missed it:** MemTxnKv does not enforce FDB transaction-size limits. - **Fix:** Reuse the existing commit-and-renew path between 90 KB fragments when a locked value exceeds `max_txn_commit_byte`. Keep the existing lock checks, pending cleanup, fragment encoding, and configuration. - **Other paths and branches:** Cover v1/v2/dual writes, load, compaction, and schema change on master. Unlocked aggregation retains its atomic overwrite behavior because it updates visible bitmaps. - **Behavior change:** One oversized locked bitmap can span metadata transactions while retaining the existing Doris publish boundary. - **Compatibility:** No protocol or persistent-format changes. ### Testing - Before the fix, v1/v2 large writes, ambiguous-commit retry, and lock-loss recovery cases failed with transaction-size errors. - ASAN: `BUILD_TYPE=ASAN ./run-cloud-ut.sh --run --filter='meta_service_test:*DeleteBitmap*' -j100`: 36 tests passed, including nine new cases. The memory store enforces transaction/KV size limits; tests verify readback, retry cleanup, explicit sub-transactions, and atomic unlocked overwrites. - Production Cloud ASAN build passed via `./build.sh --cloud -j100`, with tests and injection points disabled. - Format and `git diff --check` passed. clang-tidy was skipped at the author's request. No real-FDB or FE/BE end-to-end test was run. ### Release note Fix Cloud Merge-on-Write writes failing when a single delete bitmap exceeds the FDB transaction limit. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No. - Pre-submission status: Partially verified: unit tests and production compilation passed; real-FDB and cluster validation remain. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Implementation, tests, and this description. - Human review: Pending author/maintainer review.
bobhan1
added a commit
to bobhan1/doris
that referenced
this pull request
Nov 12, 2026
### Background Cloud Merge-on-Write stores each logical delete bitmap as a blob of 90 KB KV fragments. It already batches different bitmaps into separate metadata transactions before publishing the load version or output rowset. ### What problem does this PR solve? Issue Number: None Related PR: apache#46969 (transaction budgeting), apache#47097 (large-bitmap tests) PR Type: Bugfix Problem Summary: A single bitmap exceeding the FDB transaction limit still fails because all of its KV fragments are written in one transaction. Apply the existing transaction budget between fragments of oversized locked bitmap writes. ### Details - **Change map:** 44 changed lines in one production function, 417 lines of tests, and one test-target registration. - **Frequency:** A single large v1 bitmap or serialized v2 storage value, such as a 10,578,008-byte bitmap. - **Root cause:** The budget check runs between logical values; `blob_put` splits values into KVs within one transaction. - **Why existing tests missed it:** MemTxnKv does not enforce FDB transaction-size limits. - **Fix:** Reuse the existing commit-and-renew path between 90 KB fragments when a locked value exceeds `max_txn_commit_byte`. Keep the existing lock checks, pending cleanup, fragment encoding, and configuration. - **Other paths and branches:** Cover v1/v2/dual writes, load, compaction, and schema change on master. Unlocked aggregation retains its atomic overwrite behavior because it updates visible bitmaps. - **Behavior change:** One oversized locked bitmap can span metadata transactions while retaining the existing Doris publish boundary. - **Compatibility:** No protocol or persistent-format changes. ### Testing - Before the fix, v1/v2 large writes, ambiguous-commit retry, and lock-loss recovery cases failed with transaction-size errors. - ASAN: the related bitmap suite passed 36 tests. After adding the backend selector, the same nine new cases passed against both a private **FoundationDB 7.3.69** instance and MemTxnKv. - FDB command: `DORIS_LARGE_BITMAP_FDB_CLUSTER_FILE=/path/to/fdb.cluster BUILD_TYPE=ASAN ./run-cloud-ut.sh --run --filter='meta_service_test:LargeDeleteBitmapTest.*' --fdb='<connection-string>' -j100`. Configure the bundled 7.3.69 external-client directory for a 7.3 server; the default client API is 710. - FDB coverage: 10,578,008-byte values, v1/v2/dual readback, compaction/schema change, explicit sub-transactions, lock loss and shorter retry/tail cleanup. An ambiguous result is injected after a real commit. The atomic unlocked-overwrite case receives the actual FDB 2101 size error. GTest XML records the backend for every case; test keys in both metadata key spaces were cleaned up. - Production Cloud ASAN compilation previously passed with tests and injection points disabled; production sources are unchanged by the FDB test addition. - Format and `git diff --check` passed. clang-tidy was skipped at the author's request. No FE/BE end-to-end test was run. ### Release note Fix Cloud Merge-on-Write writes failing when a single delete bitmap exceeds the FDB transaction limit. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No. - Pre-submission status: Partially verified: unit tests against MemTxnKv and FoundationDB passed, with production compilation; no FE/BE end-to-end run. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Implementation, tests, and this description. - Human review: Pending author/maintainer review.
bobhan1
added a commit
to bobhan1/doris
that referenced
this pull request
Nov 12, 2026
### Background Cloud Merge-on-Write stores each logical delete bitmap as a blob of 90 KB KV fragments. It already batches different bitmaps into separate metadata transactions before publishing the load version or output rowset. ### What problem does this PR solve? Issue Number: None Related PR: apache#46969 (transaction budgeting), apache#47097 (large-bitmap tests) PR Type: Bugfix Problem Summary: A single bitmap exceeding the FDB transaction limit still fails because all of its KV fragments are written in one transaction. Apply the existing transaction budget between fragments of oversized locked bitmap writes. ### Details - **Change map:** 44 changed lines in one production function, 423 lines of tests, two runner lines, and one test-target registration. - **Frequency:** A single large v1 bitmap or serialized v2 storage value, such as a 10,578,008-byte bitmap. - **Root cause:** The budget check runs between logical values; `blob_put` splits values into KVs within one transaction. - **Why existing tests missed it:** MemTxnKv does not enforce FDB transaction-size limits. - **Fix:** Reuse the existing commit-and-renew path between 90 KB fragments when a locked value exceeds `max_txn_commit_byte`. Keep the existing lock checks, pending cleanup, fragment encoding, and configuration. - **Other paths and branches:** Cover v1/v2/dual writes, load, compaction, and schema change on master. Unlocked aggregation retains its atomic overwrite behavior because it updates visible bitmaps. - **Behavior change:** One oversized locked bitmap can span metadata transactions while retaining the existing Doris publish boundary. - **Compatibility:** No protocol or persistent-format changes. ### Testing - Before the fix, v1/v2 large writes, ambiguous-commit retry, and lock-loss recovery cases failed with transaction-size errors. The related ASAN bitmap suite previously passed 36 tests. - The focused suite now has 11 cases, including 32 MiB single-value readback for both v1 and v2. **All 11 passed against FoundationDB 7.3.69, and all 11 passed against MemTxnKv.** The 32 MiB FDB cases each logged six commits: one for pending metadata and five for data. - `BUILD_TYPE=ASAN ./run-cloud-ut.sh --run --filter='meta_service_test:LargeDeleteBitmapTest.*' --fdb='<connection-string>' -j100` selects FDB through the existing runner flag. Omitting `--fdb` selects MemTxnKv for this suite. No extra user environment setting is required. A 7.3 server uses the bundled 7.3.69 external client with the default API 710 client. - Coverage includes 10,578,008-byte and 32 MiB values, v1/v2/dual readback, compaction/schema change, explicit sub-transactions, lock loss and shorter retry/tail cleanup. An ambiguous result is injected after a real commit. The atomic unlocked-overwrite case receives the actual FDB 2101 size error. XML records each case's backend and the larger cases' input size; both metadata key spaces were cleaned up. - Production Cloud ASAN compilation previously passed with tests and injection points disabled; production sources are unchanged by these test/runner additions. - Format, shell syntax, and `git diff --check` passed. clang-tidy was skipped at the author's request. No FE/BE end-to-end test was run. ### Release note Fix Cloud Merge-on-Write writes failing when a single delete bitmap exceeds the FDB transaction limit. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No. - Pre-submission status: Partially verified: unit tests against MemTxnKv and FoundationDB passed, with production compilation; no FE/BE end-to-end run. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Implementation, tests, and this description. - Human review: Pending author/maintainer review.
bobhan1
added a commit
to bobhan1/doris
that referenced
this pull request
Nov 12, 2026
### Background Cloud Merge-on-Write stores each logical delete bitmap as a blob of 90 KB KV fragments. It already batches different bitmaps into separate metadata transactions before publishing the load version or output rowset. ### What problem does this PR solve? Issue Number: None Related PR: apache#46969 (transaction budgeting), apache#47097 (large-bitmap tests) PR Type: Bugfix Problem Summary: A single bitmap exceeding the FDB transaction limit still fails because all of its KV fragments are written in one transaction. Apply the existing transaction budget between fragments of oversized locked bitmap writes. ### Details - **Change map:** 44 changed lines in one production function, 423 lines of tests, and one test-target registration. - **Frequency:** A single large v1 bitmap or serialized v2 storage value, such as a 10,578,008-byte bitmap. - **Root cause:** The budget check runs between logical values; `blob_put` splits values into KVs within one transaction. - **Why existing tests missed it:** MemTxnKv does not enforce FDB transaction-size limits. - **Fix:** Reuse the existing commit-and-renew path between 90 KB fragments when a locked value exceeds `max_txn_commit_byte`. Keep the existing lock checks, pending cleanup, fragment encoding, and configuration. - **Other paths and branches:** Cover v1/v2/dual writes, load, compaction, and schema change on master. Unlocked aggregation retains its atomic overwrite behavior because it updates visible bitmaps. - **Behavior change:** One oversized locked bitmap can span metadata transactions while retaining the existing Doris publish boundary. - **Compatibility:** No protocol or persistent-format changes. ### Testing - Before the fix, v1/v2 large writes, ambiguous-commit retry, and lock-loss recovery cases failed with transaction-size errors. The related ASAN bitmap suite previously passed 36 tests. - Eleven parameterized cases are instantiated for both MemTxnKv and FdbTxnKv. **All 22 passed in one ASAN run**, including 32 MiB single-value readback for both v1 and v2 on both backends. The two larger FDB cases each logged six commits: one for pending metadata and five for data. - Command: `BUILD_TYPE=ASAN ./run-cloud-ut.sh --run --filter='meta_service_test:*LargeDeleteBitmapTest*' --fdb='<connection-string>' -j100`. Follow existing FDB tests: set `fdb_cluster_file_path` to `fdb.cluster` and construct FdbTxnKv directly. The existing `--fdb` runner option only supplies the connection file; no runner changes or extra environment variables are needed. Filters `Mem/LargeDeleteBitmapTest.*` and `Fdb/LargeDeleteBitmapTest.*` select individual backends. - Coverage includes 10,578,008-byte and 32 MiB values, v1/v2/dual readback, compaction/schema change, explicit sub-transactions, lock loss and shorter retry/tail cleanup. An ambiguous result is injected after a real commit. The atomic unlocked-overwrite case receives the actual FDB 2101 size error. XML records backend and input size; both metadata key spaces were cleaned up. FDB 7.3.69 was tested with the bundled external client and default API 710 client. - Production Cloud ASAN compilation previously passed with tests and injection points disabled; production sources are unchanged by these test revisions. - Format and `git diff --check` passed. clang-tidy was skipped at the author's request. No FE/BE end-to-end test was run. ### Release note Fix Cloud Merge-on-Write writes failing when a single delete bitmap exceeds the FDB transaction limit. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No. - Pre-submission status: Partially verified: unit tests against MemTxnKv and FoundationDB passed, with production compilation; no FE/BE end-to-end run. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Implementation, tests, and this description. - Human review: Pending author/maintainer review.
bobhan1
added a commit
to bobhan1/doris
that referenced
this pull request
Nov 12, 2026
### Background Cloud Merge-on-Write stores each logical delete bitmap as a blob of 90 KB KV fragments. It already batches different bitmaps into separate metadata transactions before publishing the load version or output rowset. ### What problem does this PR solve? Issue Number: None Related PR: apache#46969 (transaction budgeting), apache#47097 (large-bitmap tests) PR Type: Bugfix Problem Summary: A single bitmap exceeding the FDB transaction limit still fails because all of its KV fragments are written in one transaction. Apply the existing transaction budget between fragments of oversized locked bitmap writes. Extract transaction renewal, rowset checks, and pre-update removal into helpers while retaining the existing cumulative budget check between values. ### Details - **Change map:** 376 changed production lines, mostly extracting three existing blocks into helpers, 470 lines of tests, and one test-target registration. The bitmap writer shrinks from 211 to 76 lines. - **Frequency:** A single large v1 bitmap or serialized v2 storage value, such as a 10,578,008-byte bitmap. - **Root cause:** The budget check runs between logical values; `blob_put` splits values into KVs within one transaction. - **Why existing tests missed it:** MemTxnKv does not enforce FDB transaction-size limits. - **Fix:** Reuse the existing commit-and-renew path between 90 KB fragments when a locked value exceeds `max_txn_commit_byte`. Preserve lock checks, pending cleanup, fragment encoding, and configuration. Separate commit/renewal, rowset validation, and unlocked overwrite cleanup from the main writer; preserve the distinction between skipping one bitmap and stopping an obsolete aggregation request. - **Other paths and branches:** Cover v1/v2/dual writes, load, compaction, and schema change on master. Unlocked aggregation retains its atomic overwrite behavior because it updates visible bitmaps. - **Behavior change:** One oversized locked bitmap can span metadata transactions while retaining the existing Doris publish boundary. - **Compatibility:** No protocol or persistent-format changes. ### Testing - Before the fix, v1/v2 large writes, ambiguous-commit retry, and lock-loss recovery cases failed with transaction-size errors. - After extraction, ASAN bitmap tests passed **53/53**, and pre-rowset tests passed **3/3** (55 unique cases). This includes 13 parameterized cases on each of MemTxnKv and real FoundationDB 7.3.69, plus existing lock, point/range deletion, obsolete aggregation, malformed metadata, and cleanup tests. - The two-value cases send two 6 MiB values in one request: each is below the 7 MiB budget, while their sum exceeds FDB's hard limit. Both v1 and v2 read back correctly on both backends; FDB logs show two commits. Single 32 MiB v1/v2 values each use six commits, including pending metadata, and read back completely. - Commands: `BUILD_TYPE=ASAN ./run-cloud-ut.sh --run --filter='meta_service_test:*DeleteBitmap*' --fdb='<connection-string>' -j100` and the same command with `*PreRowset*`. FDB cases directly read `fdb.cluster`; the existing `--fdb` option only supplies that file. No extra environment variables or runner changes are needed. - Ambiguous results are injected after a real commit. Coverage includes dual writes, compaction/schema change, explicit sub-transactions, lock loss and shorter retry/tail cleanup, and an actual FDB 2101 size error for atomic unlocked overwrites. Both metadata key spaces were cleaned up. FDB 7.3.69 uses the bundled external client with the default API 710 client. - Production Cloud ASAN build passed via `./build.sh --cloud -j100`, with `MAKE_TEST=OFF`, injection points disabled, and Java extensions disabled. - Format and `git diff --check` passed. clang-tidy was skipped at the author's request. No FE/BE end-to-end test was run. ### Release note Fix Cloud Merge-on-Write writes failing when a single delete bitmap exceeds the FDB transaction limit. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No. - Pre-submission status: Partially verified: unit tests against MemTxnKv and FoundationDB passed, with production compilation; no FE/BE end-to-end run. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Implementation, tests, and this description. - Human review: Pending author/maintainer review.
liaoxin01
pushed a commit
that referenced
this pull request
Nov 16, 2026
…ons (#70478) ### Background Cloud Merge-on-Write stores each logical delete bitmap as a blob of 90 KB KV fragments. It already batches different bitmaps into separate metadata transactions before publishing the load version or output rowset. ### What problem does this PR solve? Issue Number: None Related PR: #46969 (transaction budgeting), #47097 (large-bitmap tests) PR Type: Bugfix Problem Summary: A single bitmap exceeding the FDB transaction limit still fails because all of its KV fragments are written in one transaction. Apply the existing transaction budget between fragments of oversized locked bitmap writes, and verify the committed prefix in the last data transaction so overlapping RPC cleanup cannot leave a successful suffix-only write. Extract transaction renewal, rowset checks, and pre-update removal into helpers while retaining the existing cumulative budget check between values. ### Details - **Change map:** 411 changed production lines, mostly extracting three existing blocks into helpers, 544 lines of tests, and one test-target registration. The bitmap writer shrinks from 211 to 85 lines; a separate 29-line helper verifies the committed prefix. - **Frequency:** A single large v1 bitmap or serialized v2 storage value, such as a 10,578,008-byte bitmap. - **Root cause:** The budget check runs between logical values; `blob_put` splits values into KVs within one transaction. - **Why existing tests missed it:** MemTxnKv does not enforce FDB transaction-size limits. Sequential retries also miss pending cleanup interleaved with fragment writes. - **Fix:** Reuse the existing commit-and-renew path between 90 KB fragments when a locked value exceeds `max_txn_commit_byte`. In the last data transaction, read the committed prefix without snapshot reads and verify every fragment's key and contents. A missing or changed prefix returns `KV_TXN_CONFLICT`; cleanup after that read conflicts with the commit. Preserve lock checks, pending cleanup, fragment encoding, and configuration. Separate commit/renewal, rowset validation, and unlocked overwrite cleanup from the main writer. - **Other paths and branches:** Cover v1/v2/dual writes, load, compaction, and schema change on master. Unlocked aggregation retains its atomic overwrite behavior because it updates visible bitmaps. - **Behavior change:** One oversized locked bitmap can span metadata transactions while retaining the existing Doris publish boundary. - **Compatibility:** No protocol or persistent-format changes. Fragmented values require one additional read of their committed prefix; other values retain their existing write path. ### Testing - Before the fix, v1/v2 large writes, ambiguous-commit retry, and lock-loss recovery cases failed with transaction-size errors. - ASAN bitmap tests passed **58**, with **1 skipped**; pre-rowset tests passed **3/3**. After deduplication, **60 passed and 1 skipped**. FoundationDB 7.3.69 passed all 16 parameterized cases; MemTxnKv passed 15 and skipped the post-validation cleanup conflict case because it does not detect read conflicts with covering range clears. Existing lock, point/range deletion, obsolete aggregation, malformed metadata, and cleanup cases also passed. - Before prefix validation, all four new cleanup/replacement test cases failed on MemTxnKv and real FDB, returning OK after prefix loss or replacement. The corrected tests cover 8 MiB and 32 MiB values in both v1 and v2, pending cleanup before validation, cleanup after validation but before commit, changed prefix contents, and successful full readback after retry. - The two-value cases send two 6 MiB values in one request: each is below the 7 MiB budget, while their sum exceeds FDB's hard limit. Both v1 and v2 read back correctly on both backends; FDB logs show two commits. Single 32 MiB v1/v2 values each use six commits, including pending metadata, and read back completely. - Commands: `BUILD_TYPE=ASAN ./run-cloud-ut.sh --run --filter='meta_service_test:*DeleteBitmap*' --fdb='<connection-string>' -j100` and the same command with `*PreRowset*`. FDB cases directly read `fdb.cluster`; the existing `--fdb` option only supplies that file. No extra environment variables or runner changes are needed. - Ambiguous results are injected after a real commit. Coverage includes dual writes, compaction/schema change, explicit sub-transactions, lock loss and shorter retry/tail cleanup, and an actual FDB 2101 size error for atomic unlocked overwrites. Both metadata key spaces were cleaned up. FDB 7.3.69 uses the bundled external client with the default API 710 client. - Production Cloud ASAN build passed via `./build.sh --cloud -j100`, with `MAKE_TEST=OFF`, injection points disabled, and Java extensions disabled. - Format and `git diff --check` passed. clang-tidy was skipped at the author's request. No FE/BE end-to-end test was run. ### Notes for Reviewers Prefix validation restores bitmap completeness at the last data commit. The pre-existing race involving a late cleanup after an RPC has already returned OK is outside this change. ### Release note Fix Cloud Merge-on-Write writes failing when a single delete bitmap exceeds the FDB transaction limit.
github-actions Bot
pushed a commit
that referenced
this pull request
Nov 16, 2026
…ons (#70478) ### Background Cloud Merge-on-Write stores each logical delete bitmap as a blob of 90 KB KV fragments. It already batches different bitmaps into separate metadata transactions before publishing the load version or output rowset. ### What problem does this PR solve? Issue Number: None Related PR: #46969 (transaction budgeting), #47097 (large-bitmap tests) PR Type: Bugfix Problem Summary: A single bitmap exceeding the FDB transaction limit still fails because all of its KV fragments are written in one transaction. Apply the existing transaction budget between fragments of oversized locked bitmap writes, and verify the committed prefix in the last data transaction so overlapping RPC cleanup cannot leave a successful suffix-only write. Extract transaction renewal, rowset checks, and pre-update removal into helpers while retaining the existing cumulative budget check between values. ### Details - **Change map:** 411 changed production lines, mostly extracting three existing blocks into helpers, 544 lines of tests, and one test-target registration. The bitmap writer shrinks from 211 to 85 lines; a separate 29-line helper verifies the committed prefix. - **Frequency:** A single large v1 bitmap or serialized v2 storage value, such as a 10,578,008-byte bitmap. - **Root cause:** The budget check runs between logical values; `blob_put` splits values into KVs within one transaction. - **Why existing tests missed it:** MemTxnKv does not enforce FDB transaction-size limits. Sequential retries also miss pending cleanup interleaved with fragment writes. - **Fix:** Reuse the existing commit-and-renew path between 90 KB fragments when a locked value exceeds `max_txn_commit_byte`. In the last data transaction, read the committed prefix without snapshot reads and verify every fragment's key and contents. A missing or changed prefix returns `KV_TXN_CONFLICT`; cleanup after that read conflicts with the commit. Preserve lock checks, pending cleanup, fragment encoding, and configuration. Separate commit/renewal, rowset validation, and unlocked overwrite cleanup from the main writer. - **Other paths and branches:** Cover v1/v2/dual writes, load, compaction, and schema change on master. Unlocked aggregation retains its atomic overwrite behavior because it updates visible bitmaps. - **Behavior change:** One oversized locked bitmap can span metadata transactions while retaining the existing Doris publish boundary. - **Compatibility:** No protocol or persistent-format changes. Fragmented values require one additional read of their committed prefix; other values retain their existing write path. ### Testing - Before the fix, v1/v2 large writes, ambiguous-commit retry, and lock-loss recovery cases failed with transaction-size errors. - ASAN bitmap tests passed **58**, with **1 skipped**; pre-rowset tests passed **3/3**. After deduplication, **60 passed and 1 skipped**. FoundationDB 7.3.69 passed all 16 parameterized cases; MemTxnKv passed 15 and skipped the post-validation cleanup conflict case because it does not detect read conflicts with covering range clears. Existing lock, point/range deletion, obsolete aggregation, malformed metadata, and cleanup cases also passed. - Before prefix validation, all four new cleanup/replacement test cases failed on MemTxnKv and real FDB, returning OK after prefix loss or replacement. The corrected tests cover 8 MiB and 32 MiB values in both v1 and v2, pending cleanup before validation, cleanup after validation but before commit, changed prefix contents, and successful full readback after retry. - The two-value cases send two 6 MiB values in one request: each is below the 7 MiB budget, while their sum exceeds FDB's hard limit. Both v1 and v2 read back correctly on both backends; FDB logs show two commits. Single 32 MiB v1/v2 values each use six commits, including pending metadata, and read back completely. - Commands: `BUILD_TYPE=ASAN ./run-cloud-ut.sh --run --filter='meta_service_test:*DeleteBitmap*' --fdb='<connection-string>' -j100` and the same command with `*PreRowset*`. FDB cases directly read `fdb.cluster`; the existing `--fdb` option only supplies that file. No extra environment variables or runner changes are needed. - Ambiguous results are injected after a real commit. Coverage includes dual writes, compaction/schema change, explicit sub-transactions, lock loss and shorter retry/tail cleanup, and an actual FDB 2101 size error for atomic unlocked overwrites. Both metadata key spaces were cleaned up. FDB 7.3.69 uses the bundled external client with the default API 710 client. - Production Cloud ASAN build passed via `./build.sh --cloud -j100`, with `MAKE_TEST=OFF`, injection points disabled, and Java extensions disabled. - Format and `git diff --check` passed. clang-tidy was skipped at the author's request. No FE/BE end-to-end test was run. ### Notes for Reviewers Prefix validation restores bitmap completeness at the last data commit. The pre-existing race involving a late cleanup after an RPC has already returned OK is outside this change. ### Release note Fix Cloud Merge-on-Write writes failing when a single delete bitmap exceeds the FDB transaction limit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fdb_txn_size is not key size+value size, actually txn size is the triple of key size plus value size at least for writing, so use txn->approximate_bytes() to calculating txn size is more accurate. However, txn->approximate_bytes() is not 100% accurate either, in my test, when txn->approximate_bytes() bingger than 8.3MB, it will meet Transaction exceeds byte limit error.
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)