Repository navigation
[fix](test) Build vparquet_writer_test against the location-based Iceberg WriteInfo - #70209
Merged
morningman merged 1 commit intoNov 4, 2026
Conversation
…berg WriteInfo ### What problem does this PR solve? Issue Number: None Related PR: apache#69392, apache#69971 Problem Summary: apache#69392 replaced the `write_path`, `original_write_path` and `target_path` fields of `IPartitionWriterBase::WriteInfo` with one `IcebergOutputLocation location`. apache#69971, merged after it, added `VParquetWriterTest.HiveAndIcebergParquetPageIndexOptions`, which still sets the old fields. Since then `doris_be_test` does not compile on master (`vparquet_writer_test.cpp:380: no matching constructor for initialization of 'VIcebergPartitionWriter'`), so the BE UT of every PR fails with BE BUILD FAIL. The case now builds its `WriteInfo` as the other Iceberg case in the same file does, `.location = IcebergOutputLocation(".", ".")`. ### Release note None ### Check List (For Author) - Test: Unit Test - `doris_be_test` builds again, and `VParquetWriterTest` passes (local ASAN build, macOS). - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 28796 ms |
Contributor
TPC-DS: Total hot run time: 150953 ms |
Contributor
ClickBench: Total hot run time: 24.5 s |
Gabriel39
approved these changes
Nov 4, 2026
bobhan1
added a commit
to bobhan1/doris
that referenced
this pull request
Nov 4, 2026
### Background Cloud MoW load initialization can synchronize rowsets from Meta Service before capturing its local rowset snapshot. The existing mutable cloud_mow_sync_rowsets_when_load_txn_begin setting controls this synchronization. ### What problem does this PR solve? Issue Number: None Related PR: apache#59754 (introduced the setting), apache#70218 (enables notifications and fixes cleanup), apache#70209 (temporary BE UT build prerequisite) PR Type: Enhancement Problem Summary: Make rowset synchronization at the start of a cloud MoW load opt-in by changing cloud_mow_sync_rowsets_when_load_txn_begin from true to false. Adjust the related CloudGroupRowsetBuilderWriterTest fixture to restore the incoming configuration value instead of hard-coding true during teardown. ### Details - **Before and after:** Cloud MoW loads no longer force sync_rowsets before constructing their initial local rowset snapshot by default. - **Configs and defaults:** cloud_mow_sync_rowsets_when_load_txn_begin changes from true to false and remains dynamically mutable. - **Impact on existing users:** An explicit true setting retains synchronization at load initialization. Notification delivery and subsequent metadata synchronization remain relevant to local rowset freshness. - **Behavior change:** Yes; skip the initial synchronization unless the setting is enabled. The test fixture preserves whichever configuration value it received. ### Testing - ./run-be-ut.sh --run --filter='CloudGroupRowsetBuilderWriterTest.*:*DeltaWriterCancelIntegrationTest.*' -j100 passed all 12 ASAN tests after this change. - The unit-test build temporarily applied apache#70209 to fix the unrelated vparquet_writer_test.cpp compilation failure. That prerequisite was removed afterward and is not included in this commit. - clang-format v16/check-format on both changed files and git diff --check passed. - The earlier 36 BE and 56 FE passing tests belong to commit 1b4d168, before this default update; they were not rerun for this commit. - clang-tidy was omitted. The stale-cache partial-update scenario below was not exercised in a running cloud cluster. ### Notes for Reviewers The stale-rowset partial-update issue identified during source review remains unresolved: if a BE has not received or synchronized a prior committed rowset, constructing the MoW context from its local snapshot can classify an existing key as new and fail a partial update under the ERROR policy or when an omitted column is non-nullable and has no default. This commit changes the default; it does not repair that path. ### Release note The BE setting cloud_mow_sync_rowsets_when_load_txn_begin now defaults to false. Cloud MoW loads skip forced rowset synchronization at initialization unless this setting is explicitly enabled. ### Check List (For Author) - Test: Unit Test; 12 BE ASAN tests passed with the prerequisite described above. - Behavior changed: Yes (see Details). - Does this need documentation: Yes; include this third default-value change in the configuration reference and release notes alongside the two notification settings. - Pre-submission status: Partially verified: focused tests passed; the known stale-cache partial-update risk remains unresolved and cloud end-to-end validation is pending. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Code, test fixture adjustment and this description. - Human review: Pending.
Contributor
Author
|
skip buildall |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
bobhan1
added a commit
to bobhan1/doris
that referenced
this pull request
Nov 6, 2026
…by default ### Background After a cloud load transaction commits in Meta Service, FE can notify the participating BEs of its visible versions. BE uses cached committed rowsets or lightweight empty-rowset entries to update local tablet metadata, with normal metadata synchronization as the fallback. ### What problem does this PR solve? Issue Number: None Related PR: apache#59754 (introduced the notification path), apache#70209 (temporary prerequisite for local BE unit-test builds) PR Type: Bugfix Problem Summary: Completed notification tasks retain their registered signatures, including when the BE feature is disabled. Consumed empty-rowset markers also remain cached because removal only covers the separate non-empty-rowset map. Release signatures on every callback exit and represent both rowset kinds in one cache so removal and expiration share the same lifecycle. Enable the existing FE and BE notification settings by default. ### Details - **Frequency:** Every accepted notification leaves its signature behind; every consumed empty entry remains until expiration. - **Root cause:** The callback does not pair task registration with removal, and the split empty/non-empty caches have different removal behavior. - **Why existing tests missed it:** They cover insertion and expiration but do not check removal of an empty entry or resubmission after a notification callback completes. - **Fix:** Add a Defer guard before the BE enable check. Store empty rowsets as null metadata in the existing committed-rowset cache, reuse insertion and expiration, and avoid a redundant lookup during cleanup. - **Other paths and branches:** The guard covers enabled notifications, disabled notifications and cache misses. Non-empty and empty committed rowsets now share cleanup; the independent MoW delete-bitmap cache is unchanged. This change targets master. - **Behavior change:** enable_notify_be_after_load_txn_commit and enable_cloud_make_rs_visible_on_be now default to true. Both retain their existing dynamic mutability. cloud_mow_sync_rowsets_when_load_txn_begin remains true. - **Concurrency and lifecycle:** Cache operations retain the existing shared mutex and acquire no new nested locks. Signature cleanup runs after tablet processing has released its locks. Refreshed entries remain protected from older expiration records. - **Compatibility and data correctness:** No RPC fields, persistent formats, transaction commit rules or partition-version visibility checks change. Missing cached data continues to use the existing metadata synchronization path. - **Observability and cost:** Move per-rowset insertion logging outside the cache lock and lower it to DEBUG as notifications become enabled by default. Existing notification and expiration logs retain transaction/tablet identifiers. ### Testing - Before the fix: four new ASAN cases ran; empty-entry removal and both enabled/disabled signature-release cases failed, while the expiration-refresh case passed. - After the final change: ./run-be-ut.sh --run --filter='CloudCommittedRSMgrTest.*:EnabledAndDisabled/CloudCommittedRowsetTaskTest.*:TaskWorkerPoolTest.*:CloudTabletApplyVisiblePendingTest.*:CloudGroupRowsetBuilderWriterTest.*' -j100 passed all 36 tests. - ./run-fe-ut.sh --run org.apache.doris.cloud.transaction.CloudGlobalTransactionMgrTest passed all 56 tests. - clang-format v16/check-format on all six changed C++ files, build-support/check-build-hygiene.sh, FE fe-common Maven validate/Checkstyle, and git diff --check passed. - BE unit-test builds used the temporary apache#70209 fix for an unrelated master compilation failure in vparquet_writer_test.cpp. The prerequisite was restored afterward and is not part of this commit. - clang-tidy was not completed. A cloud end-to-end regression and a standalone production BE build were not run. ### Release note Cloud committed-rowset notifications are enabled by default through enable_notify_be_after_load_txn_commit on FE and enable_cloud_make_rs_visible_on_be on BE. Completed notifications release their task signatures, and consumed empty-rowset entries are removed promptly. ### Check List (For Author) - Test: Unit Test; 36 BE ASAN tests and 56 FE tests passed with the prerequisite described above. - Behavior changed: Yes (see Details). - Does this need documentation: Yes; document the two default-value changes in the configuration reference and release notes. - Pre-submission status: Partially verified: targeted tests passed; cloud end-to-end regression, standalone production BE build and clang-tidy are not completed. The master BE UT build requires apache#70209. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Code, tests and this description. - Human review: Pending.
bobhan1
added a commit
to bobhan1/doris
that referenced
this pull request
Nov 6, 2026
### Background Cloud MoW load initialization can synchronize rowsets from Meta Service before capturing its local rowset snapshot. The existing mutable cloud_mow_sync_rowsets_when_load_txn_begin setting controls this synchronization. ### What problem does this PR solve? Issue Number: None Related PR: apache#59754 (introduced the setting), apache#70218 (enables notifications and fixes cleanup), apache#70209 (temporary BE UT build prerequisite) PR Type: Enhancement Problem Summary: Make rowset synchronization at the start of a cloud MoW load opt-in by changing cloud_mow_sync_rowsets_when_load_txn_begin from true to false. Adjust the related CloudGroupRowsetBuilderWriterTest fixture to restore the incoming configuration value instead of hard-coding true during teardown. ### Details - **Before and after:** Cloud MoW loads no longer force sync_rowsets before constructing their initial local rowset snapshot by default. - **Configs and defaults:** cloud_mow_sync_rowsets_when_load_txn_begin changes from true to false and remains dynamically mutable. - **Impact on existing users:** An explicit true setting retains synchronization at load initialization. Notification delivery and subsequent metadata synchronization remain relevant to local rowset freshness. - **Behavior change:** Yes; skip the initial synchronization unless the setting is enabled. The test fixture preserves whichever configuration value it received. ### Testing - ./run-be-ut.sh --run --filter='CloudGroupRowsetBuilderWriterTest.*:*DeltaWriterCancelIntegrationTest.*' -j100 passed all 12 ASAN tests after this change. - The unit-test build temporarily applied apache#70209 to fix the unrelated vparquet_writer_test.cpp compilation failure. That prerequisite was removed afterward and is not included in this commit. - clang-format v16/check-format on both changed files and git diff --check passed. - The earlier 36 BE and 56 FE passing tests belong to commit 1b4d168, before this default update; they were not rerun for this commit. - clang-tidy was omitted. The stale-cache partial-update scenario below was not exercised in a running cloud cluster. ### Notes for Reviewers The stale-rowset partial-update issue identified during source review remains unresolved: if a BE has not received or synchronized a prior committed rowset, constructing the MoW context from its local snapshot can classify an existing key as new and fail a partial update under the ERROR policy or when an omitted column is non-nullable and has no default. This commit changes the default; it does not repair that path. ### Release note The BE setting cloud_mow_sync_rowsets_when_load_txn_begin now defaults to false. Cloud MoW loads skip forced rowset synchronization at initialization unless this setting is explicitly enabled. ### Check List (For Author) - Test: Unit Test; 12 BE ASAN tests passed with the prerequisite described above. - Behavior changed: Yes (see Details). - Does this need documentation: Yes; include this third default-value change in the configuration reference and release notes alongside the two notification settings. - Pre-submission status: Partially verified: focused tests passed; the known stale-cache partial-update risk remains unresolved and cloud end-to-end validation is pending. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Code, test fixture adjustment and this description. - Human review: Pending.
bobhan1
added a commit
that referenced
this pull request
Nov 10, 2026
…faults (#70218) ### Background After a cloud load transaction commits in Meta Service, FE can notify the participating BEs of its visible versions. BE uses cached committed rowsets or lightweight empty-rowset entries to update local tablet metadata, with normal metadata synchronization as the fallback. ### What problem does this PR solve? Issue Number: None Related PR: #59754 (introduced the notification path), #70209 (fixes the unrelated Parquet unit-test build failure and is now included in master) PR Type: Bugfix Problem Summary: Completed notification tasks retain their registered signatures, including when the BE feature is disabled. Consumed empty-rowset markers also remain cached because removal only covers the separate non-empty-rowset map. Release signatures on every callback exit and represent both rowset kinds in one cache so removal and expiration share the same lifecycle. Enable the existing FE and BE notification settings by default, and default cloud_mow_sync_rowsets_when_load_txn_begin to false so cloud MoW loads skip forced synchronization when they begin. Remove the six now-redundant FE/BE overrides from the Cloud p0/p1 pipeline configs so those pipelines exercise the new defaults with unchanged effective settings. ### Details - **Frequency:** Every accepted notification leaves its signature behind; every consumed empty entry remains until expiration. - **Root cause:** The callback does not pair task registration with removal, and the split empty/non-empty caches have different removal behavior. - **Why existing tests missed it:** They cover insertion and expiration but do not check removal of an empty entry or resubmission after a notification callback completes. - **Fix:** Add a Defer guard before the BE enable check. Store empty rowsets as null metadata in the existing committed-rowset cache, reuse insertion and expiration, and avoid a redundant lookup during cleanup. - **Other paths and branches:** The guard covers enabled notifications, disabled notifications and cache misses. Non-empty and empty committed rowsets now share cleanup; the independent MoW delete-bitmap cache is unchanged. This change targets master. - **Behavior change:** `enable_notify_be_after_load_txn_commit` and `enable_cloud_make_rs_visible_on_be` now default to `true`; `cloud_mow_sync_rowsets_when_load_txn_begin` now defaults to `false`. All three settings retain their existing dynamic mutability. The cloud rowset-builder test fixture now restores the incoming setting rather than hard-coding `true` at teardown. - **Concurrency and lifecycle:** Cache operations retain the existing shared mutex and acquire no new nested locks. Signature cleanup runs after tablet processing has released its locks. Refreshed entries remain protected from older expiration records. - **Compatibility and data correctness:** RPC fields and persistent formats are unchanged. Cloud MoW loads continue to use a local flush snapshot. Skipping initialization synchronization enlarges the existing snapshot freshness window; the partial-update validation and publish-alignment algorithms are unchanged. The rolling-upgrade caveat is described below. - **Observability and cost:** Move per-rowset insertion logging outside the cache lock and lower it to DEBUG as notifications become enabled by default. Existing notification and expiration logs retain transaction/tablet identifiers. ### Testing - The implementation commits were rebased onto master `87b462e4f58104140f098f5f4a8b5d58bec70149`. The BE/FE runs below tested implementation head `17d42f6ef184c7f851920830ad3ce4861fe679aa`. Final head `4ba6d0db1480dd018a2e9353cffbb6137fe1c65d` adds only the pipeline-configuration cleanup. - On the rebased head, `./run-be-ut.sh --run --filter='CloudCommittedRSMgrTest.*:EnabledAndDisabled/CloudCommittedRowsetTaskTest.*:TaskWorkerPoolTest.*:CloudTabletApplyVisiblePendingTest.*:CloudGroupRowsetBuilderWriterTest.*:*DeltaWriterCancelIntegrationTest.*' -j100` passed all 46 ASAN tests with the requested defaults. - On the same rebased head, `./run-fe-ut.sh --run org.apache.doris.cloud.transaction.CloudGlobalTransactionMgrTest` passed all 56 tests (0 failures, errors or skips); the Maven reactor completed successfully. - The BE build compiled both `task_worker_pool_test.cpp` and `vparquet_writer_test.cpp` successfully. The latter uses the #70209 fix already merged into master; no temporary source patch was applied. - Before the original cleanup fix, the empty-entry removal test and both enabled/disabled signature-release tests failed; the expiration-refresh test passed. - `build-support/check-build-hygiene.sh` and `git diff --check` passed after rebase. The unchanged PR patch previously passed clang-format v16/check-format and FE fe-common Maven validate/Checkstyle. - For the final pipeline cleanup, checked all six removed values against the FE/BE defaults, compared the remaining parsed settings in all four files, and ran `git diff --check`: all passed. BE/FE tests were not rerun for this config-only commit because its effective pipeline values are unchanged. - clang-tidy was not rerun, as requested. A cloud end-to-end regression and a standalone production BE build were not run. The snapshot-window and mixed-version behavior below was assessed from code, without a running-cluster reproduction. ### Notes for Reviewers Local review of `4ba6d0db1480dd018a2e9353cffbb6137fe1c65d` found no Blocker or Major issue. One non-blocking P2 remains for rolling upgrades. | Existing review finding | Current disposition | | --- | --- | | [MF-001](#70218 (comment)): FE notification retries | False positive: this notification is not inserted into `AgentTaskQueue`. | | [MF-002](#70218 (comment)): stale MoW snapshot | Documented behavior tradeoff of the existing mode; the new-P1 classification is withdrawn under the intended load semantics. | | [MF-003](#70218 (comment)): old-BE signature leak | P2 remains for a new-FE/old-BE rolling-upgrade window. | | [MF-004](#70218 (comment)): protected test access | False positive: BE UT uses `-fno-access-control`, and the tests compiled and passed. | **Snapshot semantics:** Even with initialization synchronization enabled, `sync_rowsets`, local snapshot construction and the later flush are not atomic with other transactions' commits. A key absent from the captured snapshot can already enter the new-key validation path and fail under ERROR or for an omitted non-nullable column without a default. Disabling the initial synchronization can enlarge the freshness window, including while an earlier MS-committed rowset is still absent locally. This PR deliberately makes that existing mode the default; master already configures Cloud p0/p1 to use it. This is an explicit behavior/cost tradeoff, not a claim that local snapshots always contain the latest committed keys. For APPEND updates that reach publication, delete-bitmap calculation synchronizes missing versions and realigns conflicting rows from historical columns; flush-time default/null fill alone does not establish final data loss. **Rolling upgrades:** Enabling the new FE notification default while an old BE supports this task but lacks the cleanup guard exposes the old BE signature leak, including when its local notification setting is false. That process retains the signatures until it is restarted/upgraded. Keep FE notifications disabled until all receiving BEs include the cleanup fix. ### Release note Cloud committed-rowset notifications are enabled by default through enable_notify_be_after_load_txn_commit on FE and enable_cloud_make_rs_visible_on_be on BE. Completed notifications release their task signatures, and consumed empty-rowset entries are removed promptly. The BE setting cloud_mow_sync_rowsets_when_load_txn_begin now defaults to false, so cloud MoW loads skip forced rowset synchronization at initialization unless it is explicitly enabled. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: Yes; document all three default-value changes in the configuration reference and release notes. - Pre-submission status: Partially verified: the implementation passed the targeted BE/FE tests and the final pipeline cleanup passed static effective-configuration checks. Cloud end-to-end regression and a standalone production BE build were not run; clang-tidy was excluded as requested. The local review has no Blocker/Major finding, with the rolling-upgrade P2 and snapshot behavior documented above. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Code, tests and this description. - Human review: [@gavinchou previously approved implementation head `17d42f6ef18`](#70218 (review)); GitHub dismissed that approval after the pipeline cleanup commit. The final head has been reviewed locally.
bobhan1
added a commit
to bobhan1/doris
that referenced
this pull request
Nov 12, 2026
…faults (apache#70218) After a cloud load transaction commits in Meta Service, FE can notify the participating BEs of its visible versions. BE uses cached committed rowsets or lightweight empty-rowset entries to update local tablet metadata, with normal metadata synchronization as the fallback. Issue Number: None Related PR: apache#59754 (introduced the notification path), apache#70209 (fixes the unrelated Parquet unit-test build failure and is now included in master) PR Type: Bugfix Problem Summary: Completed notification tasks retain their registered signatures, including when the BE feature is disabled. Consumed empty-rowset markers also remain cached because removal only covers the separate non-empty-rowset map. Release signatures on every callback exit and represent both rowset kinds in one cache so removal and expiration share the same lifecycle. Enable the existing FE and BE notification settings by default, and default cloud_mow_sync_rowsets_when_load_txn_begin to false so cloud MoW loads skip forced synchronization when they begin. Remove the six now-redundant FE/BE overrides from the Cloud p0/p1 pipeline configs so those pipelines exercise the new defaults with unchanged effective settings. - **Frequency:** Every accepted notification leaves its signature behind; every consumed empty entry remains until expiration. - **Root cause:** The callback does not pair task registration with removal, and the split empty/non-empty caches have different removal behavior. - **Why existing tests missed it:** They cover insertion and expiration but do not check removal of an empty entry or resubmission after a notification callback completes. - **Fix:** Add a Defer guard before the BE enable check. Store empty rowsets as null metadata in the existing committed-rowset cache, reuse insertion and expiration, and avoid a redundant lookup during cleanup. - **Other paths and branches:** The guard covers enabled notifications, disabled notifications and cache misses. Non-empty and empty committed rowsets now share cleanup; the independent MoW delete-bitmap cache is unchanged. This change targets master. - **Behavior change:** `enable_notify_be_after_load_txn_commit` and `enable_cloud_make_rs_visible_on_be` now default to `true`; `cloud_mow_sync_rowsets_when_load_txn_begin` now defaults to `false`. All three settings retain their existing dynamic mutability. The cloud rowset-builder test fixture now restores the incoming setting rather than hard-coding `true` at teardown. - **Concurrency and lifecycle:** Cache operations retain the existing shared mutex and acquire no new nested locks. Signature cleanup runs after tablet processing has released its locks. Refreshed entries remain protected from older expiration records. - **Compatibility and data correctness:** RPC fields and persistent formats are unchanged. Cloud MoW loads continue to use a local flush snapshot. Skipping initialization synchronization enlarges the existing snapshot freshness window; the partial-update validation and publish-alignment algorithms are unchanged. The rolling-upgrade caveat is described below. - **Observability and cost:** Move per-rowset insertion logging outside the cache lock and lower it to DEBUG as notifications become enabled by default. Existing notification and expiration logs retain transaction/tablet identifiers. - The implementation commits were rebased onto master `87b462e4f58104140f098f5f4a8b5d58bec70149`. The BE/FE runs below tested implementation head `17d42f6ef184c7f851920830ad3ce4861fe679aa`. Final head `4ba6d0db1480dd018a2e9353cffbb6137fe1c65d` adds only the pipeline-configuration cleanup. - On the rebased head, `./run-be-ut.sh --run --filter='CloudCommittedRSMgrTest.*:EnabledAndDisabled/CloudCommittedRowsetTaskTest.*:TaskWorkerPoolTest.*:CloudTabletApplyVisiblePendingTest.*:CloudGroupRowsetBuilderWriterTest.*:*DeltaWriterCancelIntegrationTest.*' -j100` passed all 46 ASAN tests with the requested defaults. - On the same rebased head, `./run-fe-ut.sh --run org.apache.doris.cloud.transaction.CloudGlobalTransactionMgrTest` passed all 56 tests (0 failures, errors or skips); the Maven reactor completed successfully. - The BE build compiled both `task_worker_pool_test.cpp` and `vparquet_writer_test.cpp` successfully. The latter uses the apache#70209 fix already merged into master; no temporary source patch was applied. - Before the original cleanup fix, the empty-entry removal test and both enabled/disabled signature-release tests failed; the expiration-refresh test passed. - `build-support/check-build-hygiene.sh` and `git diff --check` passed after rebase. The unchanged PR patch previously passed clang-format v16/check-format and FE fe-common Maven validate/Checkstyle. - For the final pipeline cleanup, checked all six removed values against the FE/BE defaults, compared the remaining parsed settings in all four files, and ran `git diff --check`: all passed. BE/FE tests were not rerun for this config-only commit because its effective pipeline values are unchanged. - clang-tidy was not rerun, as requested. A cloud end-to-end regression and a standalone production BE build were not run. The snapshot-window and mixed-version behavior below was assessed from code, without a running-cluster reproduction. Local review of `4ba6d0db1480dd018a2e9353cffbb6137fe1c65d` found no Blocker or Major issue. One non-blocking P2 remains for rolling upgrades. | Existing review finding | Current disposition | | --- | --- | | [MF-001](apache#70218 (comment)): FE notification retries | False positive: this notification is not inserted into `AgentTaskQueue`. | | [MF-002](apache#70218 (comment)): stale MoW snapshot | Documented behavior tradeoff of the existing mode; the new-P1 classification is withdrawn under the intended load semantics. | | [MF-003](apache#70218 (comment)): old-BE signature leak | P2 remains for a new-FE/old-BE rolling-upgrade window. | | [MF-004](apache#70218 (comment)): protected test access | False positive: BE UT uses `-fno-access-control`, and the tests compiled and passed. | **Snapshot semantics:** Even with initialization synchronization enabled, `sync_rowsets`, local snapshot construction and the later flush are not atomic with other transactions' commits. A key absent from the captured snapshot can already enter the new-key validation path and fail under ERROR or for an omitted non-nullable column without a default. Disabling the initial synchronization can enlarge the freshness window, including while an earlier MS-committed rowset is still absent locally. This PR deliberately makes that existing mode the default; master already configures Cloud p0/p1 to use it. This is an explicit behavior/cost tradeoff, not a claim that local snapshots always contain the latest committed keys. For APPEND updates that reach publication, delete-bitmap calculation synchronizes missing versions and realigns conflicting rows from historical columns; flush-time default/null fill alone does not establish final data loss. **Rolling upgrades:** Enabling the new FE notification default while an old BE supports this task but lacks the cleanup guard exposes the old BE signature leak, including when its local notification setting is false. That process retains the signatures until it is restarted/upgraded. Keep FE notifications disabled until all receiving BEs include the cleanup fix. Cloud committed-rowset notifications are enabled by default through enable_notify_be_after_load_txn_commit on FE and enable_cloud_make_rs_visible_on_be on BE. Completed notifications release their task signatures, and consumed empty-rowset entries are removed promptly. The BE setting cloud_mow_sync_rowsets_when_load_txn_begin now defaults to false, so cloud MoW loads skip forced rowset synchronization at initialization unless it is explicitly enabled. - Behavior changed: Yes (see Details). - Does this need documentation: Yes; document all three default-value changes in the configuration reference and release notes. - Pre-submission status: Partially verified: the implementation passed the targeted BE/FE tests and the final pipeline cleanup passed static effective-configuration checks. Cloud end-to-end regression and a standalone production BE build were not run; clang-tidy was excluded as requested. The local review has no Blocker/Major finding, with the rolling-upgrade P2 and snapshot behavior documented above. - Tool: Codex (GPT-6). - Generated: Code, tests and this description. - Human review: [@gavinchou previously approved implementation head `17d42f6ef18`](apache#70218 (review)); GitHub dismissed that approval after the pipeline cleanup commit. The final head has been reviewed locally. (cherry picked from commit 08231e3)
bobhan1
added a commit
to bobhan1/doris
that referenced
this pull request
Nov 13, 2026
…faults (apache#70218) After a cloud load transaction commits in Meta Service, FE can notify the participating BEs of its visible versions. BE uses cached committed rowsets or lightweight empty-rowset entries to update local tablet metadata, with normal metadata synchronization as the fallback. Issue Number: None Related PR: apache#59754 (introduced the notification path), apache#70209 (fixes the unrelated Parquet unit-test build failure and is now included in master) PR Type: Bugfix Problem Summary: Completed notification tasks retain their registered signatures, including when the BE feature is disabled. Consumed empty-rowset markers also remain cached because removal only covers the separate non-empty-rowset map. Release signatures on every callback exit and represent both rowset kinds in one cache so removal and expiration share the same lifecycle. Enable the existing FE and BE notification settings by default, and default cloud_mow_sync_rowsets_when_load_txn_begin to false so cloud MoW loads skip forced synchronization when they begin. Remove the six now-redundant FE/BE overrides from the Cloud p0/p1 pipeline configs so those pipelines exercise the new defaults with unchanged effective settings. - **Frequency:** Every accepted notification leaves its signature behind; every consumed empty entry remains until expiration. - **Root cause:** The callback does not pair task registration with removal, and the split empty/non-empty caches have different removal behavior. - **Why existing tests missed it:** They cover insertion and expiration but do not check removal of an empty entry or resubmission after a notification callback completes. - **Fix:** Add a Defer guard before the BE enable check. Store empty rowsets as null metadata in the existing committed-rowset cache, reuse insertion and expiration, and avoid a redundant lookup during cleanup. - **Other paths and branches:** The guard covers enabled notifications, disabled notifications and cache misses. Non-empty and empty committed rowsets now share cleanup; the independent MoW delete-bitmap cache is unchanged. This change targets master. - **Behavior change:** `enable_notify_be_after_load_txn_commit` and `enable_cloud_make_rs_visible_on_be` now default to `true`; `cloud_mow_sync_rowsets_when_load_txn_begin` now defaults to `false`. All three settings retain their existing dynamic mutability. The cloud rowset-builder test fixture now restores the incoming setting rather than hard-coding `true` at teardown. - **Concurrency and lifecycle:** Cache operations retain the existing shared mutex and acquire no new nested locks. Signature cleanup runs after tablet processing has released its locks. Refreshed entries remain protected from older expiration records. - **Compatibility and data correctness:** RPC fields and persistent formats are unchanged. Cloud MoW loads continue to use a local flush snapshot. Skipping initialization synchronization enlarges the existing snapshot freshness window; the partial-update validation and publish-alignment algorithms are unchanged. The rolling-upgrade caveat is described below. - **Observability and cost:** Move per-rowset insertion logging outside the cache lock and lower it to DEBUG as notifications become enabled by default. Existing notification and expiration logs retain transaction/tablet identifiers. - The implementation commits were rebased onto master `87b462e4f58104140f098f5f4a8b5d58bec70149`. The BE/FE runs below tested implementation head `17d42f6ef184c7f851920830ad3ce4861fe679aa`. Final head `4ba6d0db1480dd018a2e9353cffbb6137fe1c65d` adds only the pipeline-configuration cleanup. - On the rebased head, `./run-be-ut.sh --run --filter='CloudCommittedRSMgrTest.*:EnabledAndDisabled/CloudCommittedRowsetTaskTest.*:TaskWorkerPoolTest.*:CloudTabletApplyVisiblePendingTest.*:CloudGroupRowsetBuilderWriterTest.*:*DeltaWriterCancelIntegrationTest.*' -j100` passed all 46 ASAN tests with the requested defaults. - On the same rebased head, `./run-fe-ut.sh --run org.apache.doris.cloud.transaction.CloudGlobalTransactionMgrTest` passed all 56 tests (0 failures, errors or skips); the Maven reactor completed successfully. - The BE build compiled both `task_worker_pool_test.cpp` and `vparquet_writer_test.cpp` successfully. The latter uses the apache#70209 fix already merged into master; no temporary source patch was applied. - Before the original cleanup fix, the empty-entry removal test and both enabled/disabled signature-release tests failed; the expiration-refresh test passed. - `build-support/check-build-hygiene.sh` and `git diff --check` passed after rebase. The unchanged PR patch previously passed clang-format v16/check-format and FE fe-common Maven validate/Checkstyle. - For the final pipeline cleanup, checked all six removed values against the FE/BE defaults, compared the remaining parsed settings in all four files, and ran `git diff --check`: all passed. BE/FE tests were not rerun for this config-only commit because its effective pipeline values are unchanged. - clang-tidy was not rerun, as requested. A cloud end-to-end regression and a standalone production BE build were not run. The snapshot-window and mixed-version behavior below was assessed from code, without a running-cluster reproduction. Local review of `4ba6d0db1480dd018a2e9353cffbb6137fe1c65d` found no Blocker or Major issue. One non-blocking P2 remains for rolling upgrades. | Existing review finding | Current disposition | | --- | --- | | [MF-001](apache#70218 (comment)): FE notification retries | False positive: this notification is not inserted into `AgentTaskQueue`. | | [MF-002](apache#70218 (comment)): stale MoW snapshot | Documented behavior tradeoff of the existing mode; the new-P1 classification is withdrawn under the intended load semantics. | | [MF-003](apache#70218 (comment)): old-BE signature leak | P2 remains for a new-FE/old-BE rolling-upgrade window. | | [MF-004](apache#70218 (comment)): protected test access | False positive: BE UT uses `-fno-access-control`, and the tests compiled and passed. | **Snapshot semantics:** Even with initialization synchronization enabled, `sync_rowsets`, local snapshot construction and the later flush are not atomic with other transactions' commits. A key absent from the captured snapshot can already enter the new-key validation path and fail under ERROR or for an omitted non-nullable column without a default. Disabling the initial synchronization can enlarge the freshness window, including while an earlier MS-committed rowset is still absent locally. This PR deliberately makes that existing mode the default; master already configures Cloud p0/p1 to use it. This is an explicit behavior/cost tradeoff, not a claim that local snapshots always contain the latest committed keys. For APPEND updates that reach publication, delete-bitmap calculation synchronizes missing versions and realigns conflicting rows from historical columns; flush-time default/null fill alone does not establish final data loss. **Rolling upgrades:** Enabling the new FE notification default while an old BE supports this task but lacks the cleanup guard exposes the old BE signature leak, including when its local notification setting is false. That process retains the signatures until it is restarted/upgraded. Keep FE notifications disabled until all receiving BEs include the cleanup fix. Cloud committed-rowset notifications are enabled by default through enable_notify_be_after_load_txn_commit on FE and enable_cloud_make_rs_visible_on_be on BE. Completed notifications release their task signatures, and consumed empty-rowset entries are removed promptly. The BE setting cloud_mow_sync_rowsets_when_load_txn_begin now defaults to false, so cloud MoW loads skip forced rowset synchronization at initialization unless it is explicitly enabled. - Behavior changed: Yes (see Details). - Does this need documentation: Yes; document all three default-value changes in the configuration reference and release notes. - Pre-submission status: Partially verified: the implementation passed the targeted BE/FE tests and the final pipeline cleanup passed static effective-configuration checks. Cloud end-to-end regression and a standalone production BE build were not run; clang-tidy was excluded as requested. The local review has no Blocker/Major finding, with the rolling-upgrade P2 and snapshot behavior documented above. - Tool: Codex (GPT-6). - Generated: Code, tests and this description. - Human review: [@gavinchou previously approved implementation head `17d42f6ef18`](apache#70218 (review)); GitHub dismissed that approval after the pipeline cleanup commit. The final head has been reviewed locally. (cherry picked from commit 08231e3)
bobhan1
added a commit
to bobhan1/doris
that referenced
this pull request
Nov 16, 2026
…faults (apache#70218) After a cloud load transaction commits in Meta Service, FE can notify the participating BEs of its visible versions. BE uses cached committed rowsets or lightweight empty-rowset entries to update local tablet metadata, with normal metadata synchronization as the fallback. Issue Number: None Related PR: apache#59754 (introduced the notification path), apache#70209 (fixes the unrelated Parquet unit-test build failure and is now included in master) PR Type: Bugfix Problem Summary: Completed notification tasks retain their registered signatures, including when the BE feature is disabled. Consumed empty-rowset markers also remain cached because removal only covers the separate non-empty-rowset map. Release signatures on every callback exit and represent both rowset kinds in one cache so removal and expiration share the same lifecycle. Enable the existing FE and BE notification settings by default, and default cloud_mow_sync_rowsets_when_load_txn_begin to false so cloud MoW loads skip forced synchronization when they begin. Remove the six now-redundant FE/BE overrides from the Cloud p0/p1 pipeline configs so those pipelines exercise the new defaults with unchanged effective settings. - **Frequency:** Every accepted notification leaves its signature behind; every consumed empty entry remains until expiration. - **Root cause:** The callback does not pair task registration with removal, and the split empty/non-empty caches have different removal behavior. - **Why existing tests missed it:** They cover insertion and expiration but do not check removal of an empty entry or resubmission after a notification callback completes. - **Fix:** Add a Defer guard before the BE enable check. Store empty rowsets as null metadata in the existing committed-rowset cache, reuse insertion and expiration, and avoid a redundant lookup during cleanup. - **Other paths and branches:** The guard covers enabled notifications, disabled notifications and cache misses. Non-empty and empty committed rowsets now share cleanup; the independent MoW delete-bitmap cache is unchanged. This change targets master. - **Behavior change:** `enable_notify_be_after_load_txn_commit` and `enable_cloud_make_rs_visible_on_be` now default to `true`; `cloud_mow_sync_rowsets_when_load_txn_begin` now defaults to `false`. All three settings retain their existing dynamic mutability. The cloud rowset-builder test fixture now restores the incoming setting rather than hard-coding `true` at teardown. - **Concurrency and lifecycle:** Cache operations retain the existing shared mutex and acquire no new nested locks. Signature cleanup runs after tablet processing has released its locks. Refreshed entries remain protected from older expiration records. - **Compatibility and data correctness:** RPC fields and persistent formats are unchanged. Cloud MoW loads continue to use a local flush snapshot. Skipping initialization synchronization enlarges the existing snapshot freshness window; the partial-update validation and publish-alignment algorithms are unchanged. The rolling-upgrade caveat is described below. - **Observability and cost:** Move per-rowset insertion logging outside the cache lock and lower it to DEBUG as notifications become enabled by default. Existing notification and expiration logs retain transaction/tablet identifiers. - The implementation commits were rebased onto master `87b462e4f58104140f098f5f4a8b5d58bec70149`. The BE/FE runs below tested implementation head `17d42f6ef184c7f851920830ad3ce4861fe679aa`. Final head `4ba6d0db1480dd018a2e9353cffbb6137fe1c65d` adds only the pipeline-configuration cleanup. - On the rebased head, `./run-be-ut.sh --run --filter='CloudCommittedRSMgrTest.*:EnabledAndDisabled/CloudCommittedRowsetTaskTest.*:TaskWorkerPoolTest.*:CloudTabletApplyVisiblePendingTest.*:CloudGroupRowsetBuilderWriterTest.*:*DeltaWriterCancelIntegrationTest.*' -j100` passed all 46 ASAN tests with the requested defaults. - On the same rebased head, `./run-fe-ut.sh --run org.apache.doris.cloud.transaction.CloudGlobalTransactionMgrTest` passed all 56 tests (0 failures, errors or skips); the Maven reactor completed successfully. - The BE build compiled both `task_worker_pool_test.cpp` and `vparquet_writer_test.cpp` successfully. The latter uses the apache#70209 fix already merged into master; no temporary source patch was applied. - Before the original cleanup fix, the empty-entry removal test and both enabled/disabled signature-release tests failed; the expiration-refresh test passed. - `build-support/check-build-hygiene.sh` and `git diff --check` passed after rebase. The unchanged PR patch previously passed clang-format v16/check-format and FE fe-common Maven validate/Checkstyle. - For the final pipeline cleanup, checked all six removed values against the FE/BE defaults, compared the remaining parsed settings in all four files, and ran `git diff --check`: all passed. BE/FE tests were not rerun for this config-only commit because its effective pipeline values are unchanged. - clang-tidy was not rerun, as requested. A cloud end-to-end regression and a standalone production BE build were not run. The snapshot-window and mixed-version behavior below was assessed from code, without a running-cluster reproduction. Local review of `4ba6d0db1480dd018a2e9353cffbb6137fe1c65d` found no Blocker or Major issue. One non-blocking P2 remains for rolling upgrades. | Existing review finding | Current disposition | | --- | --- | | [MF-001](apache#70218 (comment)): FE notification retries | False positive: this notification is not inserted into `AgentTaskQueue`. | | [MF-002](apache#70218 (comment)): stale MoW snapshot | Documented behavior tradeoff of the existing mode; the new-P1 classification is withdrawn under the intended load semantics. | | [MF-003](apache#70218 (comment)): old-BE signature leak | P2 remains for a new-FE/old-BE rolling-upgrade window. | | [MF-004](apache#70218 (comment)): protected test access | False positive: BE UT uses `-fno-access-control`, and the tests compiled and passed. | **Snapshot semantics:** Even with initialization synchronization enabled, `sync_rowsets`, local snapshot construction and the later flush are not atomic with other transactions' commits. A key absent from the captured snapshot can already enter the new-key validation path and fail under ERROR or for an omitted non-nullable column without a default. Disabling the initial synchronization can enlarge the freshness window, including while an earlier MS-committed rowset is still absent locally. This PR deliberately makes that existing mode the default; master already configures Cloud p0/p1 to use it. This is an explicit behavior/cost tradeoff, not a claim that local snapshots always contain the latest committed keys. For APPEND updates that reach publication, delete-bitmap calculation synchronizes missing versions and realigns conflicting rows from historical columns; flush-time default/null fill alone does not establish final data loss. **Rolling upgrades:** Enabling the new FE notification default while an old BE supports this task but lacks the cleanup guard exposes the old BE signature leak, including when its local notification setting is false. That process retains the signatures until it is restarted/upgraded. Keep FE notifications disabled until all receiving BEs include the cleanup fix. Cloud committed-rowset notifications are enabled by default through enable_notify_be_after_load_txn_commit on FE and enable_cloud_make_rs_visible_on_be on BE. Completed notifications release their task signatures, and consumed empty-rowset entries are removed promptly. The BE setting cloud_mow_sync_rowsets_when_load_txn_begin now defaults to false, so cloud MoW loads skip forced rowset synchronization at initialization unless it is explicitly enabled. - Behavior changed: Yes (see Details). - Does this need documentation: Yes; document all three default-value changes in the configuration reference and release notes. - Pre-submission status: Partially verified: the implementation passed the targeted BE/FE tests and the final pipeline cleanup passed static effective-configuration checks. Cloud end-to-end regression and a standalone production BE build were not run; clang-tidy was excluded as requested. The local review has no Blocker/Major finding, with the rolling-upgrade P2 and snapshot behavior documented above. - Tool: Codex (GPT-6). - Generated: Code, tests and this description. - Human review: [@gavinchou previously approved implementation head `17d42f6ef18`](apache#70218 (review)); GitHub dismissed that approval after the pipeline cleanup commit. The final head has been reviewed locally. (cherry picked from commit 08231e3)
morningman
added a commit
that referenced
this pull request
Nov 17, 2026
…t stream closes early (#69996) ### What problem does this PR solve? Issue Number: None Related PR: #68630, #69945 Problem Summary: **Builds on #69945, now merged.** This branch is rebased onto master `0eb5131db62` and holds seven commits: the port; its first review fix, which keeps a route past the session's next request; two fixes from the second review, which bound the BE's FE calls on stream teardown and count the BE addresses of the routes in the registry bound; and three from the third review, which drop the routes of a GetFlightInfo that fails, cancel through those routes its queries that no deferred coordinator cancels, and have the BE ask the FEs the master reaches first. #### In short This ports the FE half of branch-4.x's #68630 to master. With #69945, an Arrow Flight SQL stream that the client closes or cancels before its end cancels its query only on the BE that serves the stream, and only when that BE also holds the stream's results. A query whose results are spread over several BEs keeps running on all the other BEs until it finishes or times out. With this PR the FE remembers, for each Flight query that leaves its results on the BEs, which BEs run it; the BE that sees the aborted stream asks the FE, through the new `FrontendService.cancelFlightQuery`, to cancel the query on every one of them. #### Background - A Flight SQL query leaves its result on the BEs. With parallel result sink (the default), each BE that runs the result fragment holds one result buffer and `GetFlightInfo` returns one endpoint for it; the client reads each endpoint with its own `DoGet` stream. - Since #67504 the FE closes the coordinator as soon as `GetFlightInfo` has returned (a query that still needs it, a batch-mode external-table scan or a remote Doris scan, is kept until the session's next request). After that, nothing on the FE cancels the query on behalf of an open stream. - A `DoGet` is served by `ArrowFlightBatchLocalReader` when the result buffer is on the same BE, or by `ArrowFlightBatchRemoteReader`, which pulls the blocks from the result BE with the `fetch_arrow_data` BRPC. - #69945 makes a reader that is closed, cancelled, fails or is destroyed before EOF cancel the query on its own BE when the buffer is local, and lets an aborting `cancel_plan_fragment` free the query's Flight buffers on the BE that receives it. #### The problem, and what it cost - **An abort on one endpoint stays on that BE.** A single-fragment query has no exchange between its BEs, so cancelling it on one BE does not reach the others, and the coordinator that could have sent them a cancel is already closed. When the client aborts one endpoint (a client-side `LIMIT`, a cancelled cursor, a dropped connection), the other BEs run their pipelines to the end and keep their unread results until the query's timeout or the delayed buffer cleanup; a result that does not fit in a buffer back-pressures the result sink until the query times out. - **A remote reader cannot cancel anything.** #69945 interrupts its pending fetch, but the query on the result BE keeps running. - **A local reader whose query context has expired** frees its own buffers but cannot tell the query's other BEs. #### How this PR fixes it - **A route per Flight query on the FE.** `FlightSqlQueryCancellation` maps each result id (the fragment instance id of an endpoint) to the query id and the BRPC addresses of all the query's BEs (`Coordinator.getBackendBrpcAddresses()`, from its per-BE tasks). `StmtExecutor` registers it when the query leaves its result on the BEs (`!isReturnResultFromLocal()`), before the coordinator is closed or deferred (`FlightSqlQueryCancellation.register(ConnectContext, Coordinator)`). The route belongs to no session. The client reads the results from the BEs, so it can still hold a stream of the query open when the session's next request begins or the session is torn down, and close it early afterwards. The BE also reports an abort from its `DoGet` thread, so even a client that closes a stream before it sends its next request can be overtaken by that request. A route goes when a cancellation through it succeeds, when the GetFlightInfo that registered it fails before answering with tickets, `execution_timeout + 5 s` after it was registered, or, oldest first and as a whole, when the registry would hold more than 100,000 result ids or 1,000,000 BE addresses. A route costs about 270 bytes plus about 30 bytes per BE address, and the result ids of a query share it, so the registry stays within about 55 MB however many BEs a query runs on. Without the session as a bound, these caps keep the registry from growing with the query rate times the timeout; a query whose route was pushed out ends at its timeout, as before this PR. The registry is an insertion-ordered map under a lock, and the cancellation RPCs are sent outside it. - **`cancelFlightQuery(result_id)`, a new FrontendService RPC.** It answers NOT_FOUND when the FE has no route for the id. Otherwise it sends the existing `cancel_plan_fragment` to every BE of the query, waits at most 1 s in total for their answers, and returns OK only when all of them succeeded; on any failure the route stays, so that the caller can retry. `cancel_plan_fragment` is understood by older BEs and runs on the light work pool, not on the Flight read pool, so it also gets through when the Flight pool is saturated or shut down. - **A GetFlightInfo that fails cancels its query.** A GetFlightInfo can fail after its query ran, at the Arrow schema fetch or because a prepared statement's result schema no longer matches; no client can pull the results then. A deferred query is cancelled through its coordinator, which is still open, as before. Any other query has closed its coordinator, and nothing would cancel it before the BE's result buffer timer (`execution_timeout + 5 s`): its route now sends the cancellation to every BE of the query, without waiting for the answers, and the request's routes go. The prepared-statement check cancels its query through the executor and only drops the route. A request that fails before it begins, such as an empty query, finds the previous request's endpoints on the session and leaves their routes. - **The BE reports every abort.** After the local cleanup of #69945, `ArrowFlightBatchReaderBase::close()` calls `cancelFlightQuery` on each running FE in turn, first the FEs the master reaches and then those it reports with process UUID 0. OK ends the loop; NOT_FOUND (an FE that did not plan the query) moves on to the next FE; any other failure is retried once, and the transport is reopened after a thrift error. These calls use FE connections of their own (`ExecEnv::flight_cancel_frontend_client_cache`, at most two kept per FE), all opened with a 2 s timeout, and acquire them without backing off; an FE that cannot be connected to is not retried. A stream read to EOF never calls the FE. - The thrift method is appended to `FrontendService` with the same name, signature and comment as on branch-4.1 and branch-4.2. | Situation | Before (master with #69945) | After | |---|---|---| | Client aborts one endpoint of a query whose results are on several BEs | Only the BE holding that endpoint cancels the query; the other BEs run to the end and keep their unread results until the timeout or the delayed cleanup | The query is cancelled on every BE of the query | | Client aborts a stream served by a remote reader | The pending fetch is interrupted; the query keeps running on the result BE | The query is cancelled on the result BE and on all other BEs | | Client aborts a local stream after its query context expired | Only the local buffers are freed | The query's other BEs are cancelled too | | FE that did not plan the query is asked | — | Answers NOT_FOUND; the BE asks the next FE | | Stream read to EOF, then closed | Nothing cancelled | Nothing cancelled; the FE is not called | **Upgrades and mixed versions.** Both the FE that planned the query and the BE that serves the stream need this PR. | FE | Result BE | Effect of an abort | |---|---|---| | Without this PR | any | The BE gets an unknown-method error from `cancelFlightQuery`, logs a warning for that FE, and only the local cleanup of #69945 applies | | With this PR | With #69945 | The query is cancelled and its Flight buffers are freed on every BE | | With this PR | Without #69945 | The query's fragments are cancelled; that BE keeps its result buffer until its usual timeout | **Cost.** The FE calls run on the `DoGet` thread, including from the reader's destructor, after the local cleanup of #69945. A stopped FE stays on the BEs' list until it is dropped, reported with process UUID 0, and the FEs the master reaches are asked first, so a stopped FE delays an early close only when no reachable FE holds the route (the route expired or was pushed out, or the planning FE is itself unreachable): by about 0.1 s when it refuses connections, 3.1 s when it does not answer the connect (`thrift_connect_timeout_seconds`), and 4 s when it accepts the connection and does not answer. An FE that hangs while the master still reaches it costs up to 4 s until the master's heartbeat to it fails. A GetFlightInfo that fails sends its cancellations without waiting for them. On the FE, `cancelFlightQuery` holds one thrift thread for at most 1 s, and each idle connection a BE keeps for these calls holds another. The route keeps only ids and addresses, not the coordinator, its scan state or its query queue slot, and the registry holds at most 100,000 result ids and 1,000,000 BE addresses, about 55 MB. Differences from branch-4.x: the route is registered in master's `!isReturnResultFromLocal()` branch (#67900) instead of the `ARROW_FLIGHT_SQL` connect type branch, before the `mustOutliveDispatch()` deferral, so deferred queries are routed as well; a route does not belong to the session: branch-4.x drops it when the session's next request begins and when the session is torn down (`FlightSqlChannel.reset()`), so an abort reported after either finds no route and the other BEs run until the timeout, while here a route stays until its cancellation succeeds or it expires, and the registry is bounded instead; the BE reader includes `runtime/frontend_info.h` itself, because `exec_env.h` no longer includes it (#66400); the BE test reads and restores `ExecEnv::_frontends` through its `unique_ptr`; the FE test is JUnit 5 and tests that a route outlives the session's next request and its teardown, and the bound; the regression suite keeps its table at the end, as AGENTS.md asks; the two branches of the BE's FE loop got comments; the BE asks the FEs the master reaches first, where branch-4.x asks them in address order; and a GetFlightInfo that fails cancels through their routes its queries that were not deferred, where branch-4.x only finalizes the deferred executors and cancels nothing. #### Results - FE UT `FlightSqlQueryCancellationTest` (9 tests, new): a cancellation reaches every BE of the route after the coordinator and the selected endpoint's fragment are gone, and a successful one removes the route for all of its result ids; a BE that answers an error keeps the route, and the retry succeeds; routes expire `execution_timeout + 5 s` after registration; a route registered from a session survives the session's next request (`beginRequest()`) and its teardown (`releaseProtocolSession()`) and still cancels; the bound drops whole routes, oldest first, also when the BE addresses exceed it; a route whose result id a newer registration took over releases its addresses; an expired route leaves at the next registration; a request that fails before answering with tickets drops its routes as a whole, and cancels the queries no deferred coordinator cancelled on all their BEs without waiting for them. - `DorisFlightSqlProducerTest`: a GetFlightInfo whose schema fetch fails cancels its query on both its BEs and drops its route; a deferred query is cancelled once, through its coordinator; an empty query leaves the previous request's route and cancels nothing. `DorisFlightSqlSchemaTest#executionSchemaDriftExpiresHandleAndCleansResults` now also requires the route of a prepared statement whose result schema changed to go. - `StmtExecutorTest#testCloudReplanDispatchProfileLifecycle` now checks, for its two Flight outcomes (closed and deferred coordinator), that the dispatched query registered its route; its coordinator mock returns the query id and one BE address, and its `exec()` adds one endpoint, as the coordinator does. - BE UT `ArrowFlightRemoteCancelTest` gets the FE half of branch-4.x's fixture back: an in-process thrift FE (`FlightCancelCoordinator`) that answers `cancelFlightQuery` by sending `cancel_plan_fragment` to an in-process BRPC result BE. The 9 new tests: a remote reader's close reaches the result BE; the cancellation also works when the Flight work pool rejects work; an FE error is retried once; an older result BE that only understands query cancellation gets it and keeps its buffer; a local reader whose query context has expired still reports the abort; a stream read to EOF does not call the FE; a close is not held up by a 60 s transport that another caller left in the shared FE cache to an FE whose first answer stalls, moves past an FE that refuses connections, and asks an FE that accepts connections and never answers, reported with process UUID 0 and sorting first, only after the planning FE (14 ms; 4.1 s in address order). `CancellationInterruptsPendingBrpcFetch` and `FetchAfterCancellationStillCompletesRpc` are back to branch-4.x's form, which needs the FE route (#69945 carried variants without it). - 28 Flight session and coordinator FE test classes (297 tests) pass, with Checkstyle. The 4 BE suites (`ArrowFlightCancelTest`, `ArrowFlightRemoteCancelTest`, `ResultBufferMgrTest`, `ArrowResultBlockBufferTest`, 30 tests) pass 10 times in a row in one ASAN process. - Mutations, each red in its target tests: on the FE, a failed cancellation drops the route; a successful one keeps it; routes never expire; `StmtExecutor` does not register; and, for the review fix, the session drops its routes again at its next request and teardown; the bound evicts the route just registered instead of the oldest one, or single result ids instead of whole routes; an expired route can still be cancelled; registration keeps expired routes. On the BE, the reader stops calling the FEs (the 7 tests that need the route turn red, the other 11 stay green); the reader does not retry. For the second review's fixes: on the BE, taking the connections from the shared cache and acquiring them with the default backoff (one round) turn the two new tests red, each its own (the close took 5.0 s and 7.4 s); with the reader as it was before these fixes they took 5.0 s and 14.7 s. On the FE, not releasing a route's addresses, releasing them per result id instead of with the last one, leaving the address bound out of the eviction, and keeping the addresses of the route an id leaves when a newer registration takes it over each turn their target test red. That an FE which cannot be connected to gets no second attempt has no test of its own: against an FE that refuses connections the second attempt costs about 0.1 s, and the difference only shows with an FE that does not answer the connect (3.1 s instead of 6.2 s). For the third review's fixes: on the FE, dropping the routes whether or not the request has begun, dropping single result ids instead of whole routes, cancelling deferred queries through their routes too, dropping the routes without cancelling, not handling the failure at all, waiting for the BEs' answers, and leaving the prepared-statement branch's route each turn their target tests red; on the BE, asking the FEs in address order turns the new test red (4.1 s). - On a local single-BE cluster, `arrow_flight_sql_p0/test_flight_cancel_cleanup` (new; it aborts one endpoint after the first batch, by closing it, by an explicit cancel, and by closing it after the session ran its next request, and each time requires the query's pipelines to disappear from every BE within 20 s) passes, and the BE log shows the `cancel_plan_fragment` that the FE sent through the route for all three aborts. Before the review fix it showed it for one of the three: the third abort found no route, and so did the first, whose report came after the suite's next request (its first attempt met a stale connection after an FE restart; the retry reopened it and found no route). The other 20 `arrow_flight_sql_p0` suites and the 14 `external_table_p0/remote_doris` suites pass. A single BE cannot show the cross-BE effect: there the BE serving the stream already cancels the query itself (#69945), so the suite passes without this PR too; on several BEs it needs this PR. Locally the cross-BE path is covered by the BE UT's in-process FE and remote result BE. - Rebased onto master `0eb5131db62`, after #69830 removed `NereidsCoordinator` and made its implementation the `Coordinator`: `getBackendBrpcAddresses()` is now a method of that class, with the body of the former `NereidsCoordinator` override (the legacy implementation went with the old class). `git range-diff` shows no other change to the first commit and none to the second. The FE and BE tests above ran on this base; master's `doris_be_test` does not build since #69971 (`vparquet_writer_test.cpp`), so the BE UT here ran with that test's one-line fix (#70209) applied locally. The regression suites were not rerun on this base, including the `arrow_flight_sql_p0` suites master added since `1adeb3a6801`. Co-authored-by: Gabriel <liwenqiang@selectdb.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #69392, #69971
Problem Summary:
In short.
doris_be_testdoes not compile on master since #69971 was merged, so the BE UT of every PR fails withBE BUILD FAIL. This fixes the one test case that breaks it.The problem. #69392 replaced the
write_path,original_write_pathandtarget_pathfields ofIPartitionWriterBase::WriteInfowith oneIcebergOutputLocation location, and updated the callers it knew about. #69971, merged after it, addedVParquetWriterTest.HiveAndIcebergParquetPageIndexOptions, which still sets the old fields:Each PR built alone, so both were green. TeamCity's BE UT has failed this way for PRs merged with master
0eb5131db62(for example build 1092024 of #69605).How this PR fixes it. The case builds its
WriteInfoas the other Iceberg case in the same file does,.location = IcebergOutputLocation(".", "."). Nothing else changes.Release note
None
Check List (For Author)
Test
doris_be_testbuilds again with this change, andVParquetWriterTest(16 tests) passes, on a local ASAN build (macOS).Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)