Skip to content

[fix](cloud) Fix infinite loop and OOM in FDB conflict logging - #70417

Merged
wyxxxcat merged 1 commit into
apache:masterfrom
wyxxxcat:codex/fix-fdb-conflict-range-pagination-master
Nov 17, 2026
Merged

wyxxxcat merged 1 commit into
apache:masterfrom
wyxxxcat:codex/fix-fdb-conflict-range-pagination-master

Conversation

@wyxxxcat

Copy link
Copy Markdown
Collaborator

Background

When an FDB commit fails with a transaction conflict, Meta Service can log the conflicting keys and the transaction's read and write conflict ranges. These diagnostics read FDB special key ranges when enable_logging_conflict_keys is enabled.

What problem does this PR solve?

Issue Number: None
Related PR: None
PR Type: Bugfix

Problem Summary:

Conflict logging can enter an infinite loop when a diagnostic range spans multiple pages. The loop repeatedly reads the same completed FDB future and appends the same key/value pairs, causing unbounded memory growth and potentially an MS OOM. Read all three diagnostic ranges through the existing full-range iterator so each page advances on the same transaction. Keep full keys for paging and strip the special key prefix only from copied results.

Details

  • Frequency: Triggered when conflict logging is enabled, an FDB commit conflicts, and a diagnostic range returns more=true.
  • Root cause: Each getter creates one range-read future outside its loop. fdb_future_get_keyvalue_array only extracts that future's result; it does not fetch the next page. Repeating it leaves more=true and keeps copying the same page into the result vector.
  • Why existing tests missed it: The existing test uses one conflicting key and reads both range ends in one page.
  • Fix: Share one helper backed by full_range_get across the conflicting-key, read-conflict, and write-conflict getters. Reuse its full-key cursor and future ownership, copy each result before advancing, and return iterator errors. The original commit error remains unchanged.
  • Other paths and branches: All three conflict diagnostic getters use the same fix. No backports are included.
  • Behavior change: Paginated conflict diagnostics finish reading their results or return a read error instead of looping over one page until memory is exhausted.

Testing

  • Extended TxnKvTest.ReportConflictingRange to force one key/value pair per page for all three getters and compare the complete results with a single-page read. The test also covers an embedded zero byte in the key and a timeout on the second page.
  • Static review only. The updated tests have not been run. Compilation and formatting checks were not run, as requested by the author.

Release note

Fix a possible Meta Service OOM caused by an infinite loop while logging paginated FDB conflict ranges with enable_logging_conflict_keys enabled.

Check List (For Author)

  • Behavior changed: Yes (see Details)
  • Does this need documentation: No
  • Pre-submission status: Partially verified: static review only; compilation, formatting checks, and unit tests have not been run.

AI Disclosure

  • Tool: Codex (GPT-6)
  • Generated: Code, test changes, and this description.
  • Human review: Not confirmed.

Background

What problem does this PR solve?

Issue Number:
Related PR:
PR Type:

Problem Summary:

Details

Testing

Notes for Reviewers

Release note

Check List (For Author)

  • Behavior changed:
  • Does this need documentation:
  • Pre-submission status:

AI Disclosure

  • Tool:
  • Generated:
  • Human review:

@wyxxxcat
wyxxxcat force-pushed the codex/fix-fdb-conflict-range-pagination-master branch 2 times, most recently from 6a04b57 to e2b5b69 Compare November 10, 2026 03:11
@wyxxxcat

Copy link
Copy Markdown
Collaborator Author

run buildall

@wyxxxcat
wyxxxcat force-pushed the codex/fix-fdb-conflict-range-pagination-master branch from e2b5b69 to ea44bcb Compare November 15, 2026 14:00
@wyxxxcat

Copy link
Copy Markdown
Collaborator Author

run buildall

When an FDB commit fails with a transaction conflict, Meta Service can log the conflicting keys and the transaction's read and write conflict ranges. These diagnostics read FDB special key ranges when enable_logging_conflict_keys is enabled.

Issue Number: None
Related PR: None
PR Type: Bugfix

Problem Summary:

Conflict logging can enter an infinite loop when a diagnostic range spans multiple pages. The loop repeatedly reads the same completed FDB future and appends the same key/value pairs, causing unbounded memory growth and potentially an MS OOM. Read all three diagnostic ranges through the existing full-range iterator so each page advances on the same transaction. Keep full keys for paging and strip the special key prefix only from copied results.

- **Frequency:** Triggered when conflict logging is enabled, an FDB commit conflicts, and a diagnostic range returns more=true.
- **Root cause:** Each getter creates one range-read future outside its loop. fdb_future_get_keyvalue_array only extracts that future's result; it does not fetch the next page. Repeating it leaves more=true and keeps copying the same page into the result vector.
- **Why existing tests missed it:** The existing test uses one conflicting key and reads both range ends in one page.
- **Fix:** Share one helper backed by full_range_get across the conflicting-key, read-conflict, and write-conflict getters. Reuse its full-key cursor and future ownership, copy each result before advancing, and return iterator errors. The original commit error remains unchanged.
- **Other paths and branches:** All three conflict diagnostic getters use the same fix. No backports are included.
- **Behavior change:** Paginated conflict diagnostics finish reading their results or return a read error instead of looping over one page until memory is exhausted.

- Extended TxnKvTest.ReportConflictingRange to force one key/value pair per page for all three getters and compare the complete results with a single-page read. The test also covers an embedded zero byte in the key and a timeout on the second page.
- Static review only. The updated tests have not been run. Compilation and formatting checks were not run, as requested by the author.

Fix a possible Meta Service OOM caused by an infinite loop while logging paginated FDB conflict ranges with enable_logging_conflict_keys enabled.

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Partially verified: static review only; compilation, formatting checks, and unit tests have not been run.

- Tool: Codex (GPT-6)
- Generated: Code, test changes, and this description.
- Human review: Not confirmed.
@wyxxxcat
wyxxxcat force-pushed the codex/fix-fdb-conflict-range-pagination-master branch from ea44bcb to 5a3506a Compare November 16, 2026 02:24
@wyxxxcat

Copy link
Copy Markdown
Collaborator Author

run buildall

@wyxxxcat

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step.
Workflow run: https://github.com/apache/doris/actions/runs/43833358602

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@wyxxxcat

Copy link
Copy Markdown
Collaborator Author

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#70417
commit: 5a3506ae2ed32f399e259236f1ba23028f9ae870
base: eabe0515aafb833b7ebba419bab6ed5a34db8641
reviewed_at: 2026-11-17T16:22+08:00
reviewer: wyxxxcat
model: gpt-6-astra
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

Notes for maintainers

  • Static review only: builds, unit tests, formatting checks, and benchmarks were not run. The added TxnKvTest.ReportConflictingRange coverage at cloud/test/txn_kv_test.cpp:1618-1640 remains execution-unverified.
  • Per the requester, contrib/ third-party code was excluded; neither changed file is in that directory.

Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.

@luwei16
luwei16 requested a review from gavinchou November 17, 2026 09:33
@wyxxxcat
wyxxxcat merged commit 52e7be0 into apache:master Nov 17, 2026
36 of 37 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Nov 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

github-actions Bot pushed a commit that referenced this pull request Nov 17, 2026
### Background

When an FDB commit fails with a transaction conflict, Meta Service can
log the conflicting keys and the transaction's read and write conflict
ranges. These diagnostics read FDB special key ranges when
enable_logging_conflict_keys is enabled.

### What problem does this PR solve?

Issue Number: None
Related PR: None
PR Type: Bugfix

Problem Summary:

Conflict logging can enter an infinite loop when a diagnostic range
spans multiple pages. The loop repeatedly reads the same completed FDB
future and appends the same key/value pairs, causing unbounded memory
growth and potentially an MS OOM. Read all three diagnostic ranges
through the existing full-range iterator so each page advances on the
same transaction. Keep full keys for paging and strip the special key
prefix only from copied results.

### Details

- **Frequency:** Triggered when conflict logging is enabled, an FDB
commit conflicts, and a diagnostic range returns more=true.
- **Root cause:** Each getter creates one range-read future outside its
loop. fdb_future_get_keyvalue_array only extracts that future's result;
it does not fetch the next page. Repeating it leaves more=true and keeps
copying the same page into the result vector.
- **Why existing tests missed it:** The existing test uses one
conflicting key and reads both range ends in one page.
- **Fix:** Share one helper backed by full_range_get across the
conflicting-key, read-conflict, and write-conflict getters. Reuse its
full-key cursor and future ownership, copy each result before advancing,
and return iterator errors. The original commit error remains unchanged.
- **Other paths and branches:** All three conflict diagnostic getters
use the same fix. No backports are included.
- **Behavior change:** Paginated conflict diagnostics finish reading
their results or return a read error instead of looping over one page
until memory is exhausted.

### Testing

- Extended TxnKvTest.ReportConflictingRange to force one key/value pair
per page for all three getters and compare the complete results with a
single-page read. The test also covers an embedded zero byte in the key
and a timeout on the second page.
- Static review only. The updated tests have not been run. Compilation
and formatting checks were not run, as requested by the author.

### Release note

Fix a possible Meta Service OOM caused by an infinite loop while logging
paginated FDB conflict ranges with enable_logging_conflict_keys enabled.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Partially verified: static review only;
compilation, formatting checks, and unit tests have not been run.

### AI Disclosure

- Tool: Codex (GPT-6)
- Generated: Code, test changes, and this description.
- Human review: Not confirmed.

### Background

<!-- How the feature or module this PR touches works today, in two to
four plain sentences, for a reviewer who doesn't know it. Link the user
doc if there is one. Leave the problem and the fix to Problem Summary.
-->

### What problem does this PR solve?

<!-- Issue Number: "close #N" if this PR resolves the issue, "#N" if it
only relates to it, "None" if there is none. A Feature needs an issue.
-->
<!-- Related PR: "#N (relation)" for each PR a reviewer should know
about, such as "#123 (introduced the bug)", separated by commas. Write
"None" if no related PR is known. -->
<!-- PR Type: Bugfix, Performance, Feature, Enhancement, Refactor, Test,
Build/CI/Chore, Revert, Release/Dependency, or Other. -->

Issue Number:
Related PR:
PR Type:

Problem Summary:

<!-- Two to five sentences: the problem or need and who is affected; the
approach in one sentence; when the diff is larger than that suggests,
what else it had to handle, or that most of it is tests; and what
existing users see after an upgrade. -->

### Details

<!--
For a diff of more than about 300 changed lines, start with "- **Change
map:**": how many lines are production code, tests, generated files, and
docs, then the code in parts, each with the problem it handles and its
size.
Then answer each field of your PR type on its own line, as "- **Field:**
answer", in this order:
Bugfix: Frequency, Root cause, Why existing tests missed it, Fix, Other
paths and branches
Performance: Why it was slow, Optimization, Results, Applicability and
cost
  Feature: Scope, Design, Usage, Default behavior, Compatibility
Enhancement: Before and after, Configs and defaults, Impact on existing
users
Refactor: Moves vs. logic changes, Behavior change, Compatibility,
Performance impact, Rollback
Test, adding tests: Scenarios, What existing tests miss, Fails when
broken
  Test, fixing a test: Failure, Cause, Stability evidence
Build/CI/Chore: Affected, Developer and release impact, License and
third-party, Rollback
  Revert: Why not fix forward, Original author notified, Follow-up
Release/Dependency: Versions, Upstream changes, Compatibility, Rollback
  Other: Why Other
"None" is a valid answer when it is true. Don't restate the diff.
-->

### Testing

<!-- The tests, suites, or commands that ran, what each shows, and where
it ran (locally or in CI). For a fix, the test that fails before it and
passes after. -->

### Notes for Reviewers

<!-- Optional; delete this section if there is nothing to say. Why each
non-obvious edit was made, what you are unsure about, and where to start
reading if Details has no Change map. -->

### Release note

<!-- One paragraph on what users will notice, or None. -->

### Check List (For Author)

<!-- Behavior changed: "No", or "Yes (see Details)" with a Behavior
change field in Details. -->
<!-- Does this need documentation: "No", or "Yes" with the
apache/doris-website PR or the plan. -->
<!-- Pre-submission status: "Ready"; "Partially verified: " and what is
not verified; or "Not ready: " and why, for a Draft PR. -->

- Behavior changed:
- Does this need documentation:
- Pre-submission status:

### AI Disclosure

<!-- Tool: the agent and model, or None. Generated: the parts it wrote
(code, tests, this description), or None. Human review: what you
reviewed of the generated parts, or "Not applicable" without AI tools.
You may add "- Prompts:" to summarize how you directed the agent. -->

- Tool:
- Generated:
- Human review:
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.x dev/4.1.x dev/4.2.x reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants