Repository navigation
[fix](fe) Report active compute group in processlist - #68111
Merged
Merged
Conversation
Issue Number: None
Related PR: None
Problem Summary: SHOW PROCESSLIST read the cached policy-selected compute group directly, while query routing gives precedence to the session compute group. When a session or per-query hint selected another compute group, the query ran there but PROCESSLIST reported the stale cached group. Prefer the active session compute group when rendering process rows and retain the cached value as the fallback.
SHOW PROCESSLIST now reports the compute group used by the running query.
- Test: Unit Test
- ./run-fe-ut.sh --run org.apache.doris.qe.ConnectContextTest
- Behavior changed: Yes. The CloudCluster column now reports the active session compute group instead of a stale cached value.
- Does this need documentation: No
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Contributor
There was a problem hiding this comment.
Static review outcome: request changes for PROCESSLIST diagnostic correctness.
The direct precedence test covers only a same-thread field read. The implementation can still report a value that did not route the displayed query: it exposes the variable in non-cloud mode, and in cloud mode it reads mutable session state rather than a stable statement-scoped routing identity.
Checkpoint conclusions:
- Goal and tests: incomplete. The ordinary direct cloud precedence works, but non-cloud, prepared/multi-statement, temporary SessionVariable replacement, and Arrow Flight lifecycles are not correct or covered.
- Scope and compatibility: the code change is small and wire/storage compatible, but the new behavior is incorrectly broadened to shared-nothing mode. No FE-BE schema, persisted format, or symbol change was found.
- Concurrency and lifecycle: local and all-FE/RPC PROCESSLIST share this renderer. Prepared execution lacks publication for a late SET_VAR write; view analysis safely publishes a replacement object that drops the group; Flight reverts SET_VAR while the query is still displayed during schema fetch.
- Parallel paths and special conditions: ordinary forwarding, persistent USE/SET, single-statement COM_QUERY, and null/empty fallback were traced without another distinct defect. Post-close deferred Flight work remains the pre-existing idle-session model.
- Configuration, persistence, transactions, and data writes: not applicable; no related change was found.
- Performance and observability: direct cost is negligible, but the affected column is itself the observability surface and is wrong in the cases above. No additional metric is required once the snapshot is correct.
- Test quality: the added JUnit test has the right column index but does not enable cloud mode or exercise routing, concurrency, replacement, local/RPC collection, or Flight lifecycle.
- PR metadata: the body text says Unit Test and behavior changed, while the template boxes say Regression test and behavior unchanged; the release note also says None despite a user-visible PROCESSLIST change.
No additional user focus was supplied. Per the review contract, this was a static-only review; no build or tests were run locally.
Contributor
TPC-H: Total hot run time: 27998 ms |
Contributor
TPC-DS: Total hot run time: 153470 ms |
Contributor
ClickBench: Total hot run time: 24.09 s |
gavinchou
approved these changes
Sep 17, 2026
gavinchou
approved these changes
Sep 17, 2026
morrySnow
approved these changes
Sep 18, 2026
github-actions Bot
pushed a commit
that referenced
this pull request
Sep 18, 2026
Problem Summary: due to #50783 ,SHOW PROCESSLIST read the cached policy-selected compute group directly, while query routing gives precedence to the session compute group. When a session or per-query hint selected another compute group, the query ran there but PROCESSLIST reported the stale cached group. Prefer the active session compute group when rendering process rows and retain the cached value as the fallback. SHOW PROCESSLIST now reports the compute group used by the running query. - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.qe.ConnectContextTest - Behavior changed: Yes. The CloudCluster column now reports the active session compute group instead of a stale cached value.
yiguolei
pushed a commit
that referenced
this pull request
Sep 19, 2026
Problem Summary: due to #50783 ,SHOW PROCESSLIST read the cached policy-selected compute group directly, while query routing gives precedence to the session compute group. When a session or per-query hint selected another compute group, the query ran there but PROCESSLIST reported the stale cached group. Prefer the active session compute group when rendering process rows and retain the cached value as the fallback. SHOW PROCESSLIST now reports the compute group used by the running query. - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.qe.ConnectContextTest - Behavior changed: Yes. The CloudCluster column now reports the active session compute group instead of a stale cached value.
yiguolei
pushed a commit
that referenced
this pull request
Sep 20, 2026
Problem Summary: due to #50783 ,SHOW PROCESSLIST read the cached policy-selected compute group directly, while query routing gives precedence to the session compute group. When a session or per-query hint selected another compute group, the query ran there but PROCESSLIST reported the stale cached group. Prefer the active session compute group when rendering process rows and retain the cached value as the fallback. SHOW PROCESSLIST now reports the compute group used by the running query. - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.qe.ConnectContextTest - Behavior changed: Yes. The CloudCluster column now reports the active session compute group instead of a stale cached value.
yiguolei
pushed a commit
that referenced
this pull request
Sep 20, 2026
Problem Summary: due to #50783 ,SHOW PROCESSLIST read the cached policy-selected compute group directly, while query routing gives precedence to the session compute group. When a session or per-query hint selected another compute group, the query ran there but PROCESSLIST reported the stale cached group. Prefer the active session compute group when rendering process rows and retain the cached value as the fallback. SHOW PROCESSLIST now reports the compute group used by the running query. - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.qe.ConnectContextTest - Behavior changed: Yes. The CloudCluster column now reports the active session compute group instead of a stale cached value.
16 tasks
yiguolei
pushed a commit
that referenced
this pull request
Oct 14, 2026
pick from #68111 Problem Summary: due to #50783 ,SHOW PROCESSLIST read the cached policy-selected compute group directly, while query routing gives precedence to the session compute group. When a session or per-query hint selected another compute group, the query ran there but PROCESSLIST reported the stale cached group. Prefer the active session compute group when rendering process rows and retain the cached value as the fallback. SHOW PROCESSLIST now reports the compute group used by the running query. - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.qe.ConnectContextTest - Behavior changed: Yes. The CloudCluster column now reports the active session compute group instead of a stale cached value. (cherry picked from commit 102f697) ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
yiguolei
pushed a commit
that referenced
this pull request
Oct 15, 2026
pick from #68111 Problem Summary: due to #50783 ,SHOW PROCESSLIST read the cached policy-selected compute group directly, while query routing gives precedence to the session compute group. When a session or per-query hint selected another compute group, the query ran there but PROCESSLIST reported the stale cached group. Prefer the active session compute group when rendering process rows and retain the cached value as the fallback. SHOW PROCESSLIST now reports the compute group used by the running query. - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.qe.ConnectContextTest - Behavior changed: Yes. The CloudCluster column now reports the active session compute group instead of a stale cached value. (cherry picked from commit 102f697) ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
4 of 12 tasks
morningman
added a commit
that referenced
this pull request
Nov 2, 2026
…async MV (#69749) ### What problem does this PR solve? Issue Number: None Related PR: #67010 Problem Summary: **In short.** This ports branch-4.1 #67010 (also on branch-4.2) to master: CREATE / ALTER ROUTINE LOAD and async materialized views accept a `compute_group` property in cloud mode, so a background job can be pinned to a compute group instead of inheriting one implicitly, and the declared group is re-checked before every task. **Background.** In cloud mode every query runs in a compute group. A routine load job snapshots the compute group of the session that created it, and an async MV has no compute group of its own: an automatic refresh runs wherever `admin` resolves to, and a manual `REFRESH` borrows the group of the session that triggered it. **The problem, and what it cost.** On master nothing lets a user choose where a background job runs: | | master today | with this PR | |---|---|---| | Routine load | runs in whatever group the creating session was on; ALTER cannot move it | `"compute_group" = "cg_etl"` on CREATE or ALTER pins it | | Async MV, automatic refresh | runs where `admin` resolves to | runs in the declared group | | Async MV, manual `REFRESH` | runs in the triggering session's group, so the same MV refreshes in different places depending on who triggered it | runs in the declared group; an MV that declares none keeps borrowing the session's group | | Async MV, `REFRESH ... WITH DRY RUN` and `EXPLAIN REFRESH` | run and plan where `admin` resolves to | run and plan in the declared group; an MV that declares none resolves as today | | A pinned group is dropped, or the owner's USAGE on it is revoked | (no pinning) | the next task fails with that reason instead of a later "no available BE found" | | `compute_group` in non-cloud mode, or `DEFAULT` | (no property) | rejected at CREATE / ALTER | Four smaller defects go with it on master: CREATE ROUTINE LOAD checks the `workload_group` in the session's compute group namespace although the job runs elsewhere; `RoutineLoadJob.plan()` builds the sink's plan context from the raw snapshot field rather than the effective group; and the statement re-parsed when a routine load job is loaded from metadata resolves its `workload_group` again. On image load that lookup runs before any workload group exists (`PersistMetaModules` reads `routineLoadJobs` before `workloadGroups`, and until then the Env holds the empty `WorkloadGroupMgr` its constructor created), and a failure there sets the job to the final `CANCELLED`, so a job created with `workload_group` is cancelled by the first restart that loads it from an image (shown by a unit test with an empty manager, not reproduced on a cluster). And when the partition refresh fails, which `RoutineLoadJob.update()` runs before it takes the job lock, it pauses a NEED_SCHEDULE job without that lock, so a STOP that completes in between is turned back into PAUSED on the master while replay keeps STOPPED (Kafka metadata failures and Kinesis shard listing alike). **How this PR fixes it.** Same design as #67010: - The declaration is stored in the existing property maps (`RoutineLoadJob.jobProperties`, `MTMV.mvProperties`) and written only when declared: no new persisted field, journal type or meta version, and an absent key keeps today's implicit resolution, which is also how a later binding model tells a pinned group from an inherited one. `DEFAULT` is rejected because that model reserves it for "follow the owner's default group"; non-cloud mode rejects the property, and `ConnectContext.setCloudCluster()` is a no-op outside cloud mode. - `ComputeGroupBindingUtil` holds the two checks: `validateDeclaredComputeGroup` at CREATE / ALTER (USAGE for the current user, then existence, the order of `USE @<group>`), and `checkComputeGroupBeforeTask` before every task, against the identity the task runs as. A missing group keeps `INTERNAL_ERR`, so a routine load job is paused retryably and resumes by itself when the group comes back (a group scaled to zero backends leaves the cluster map and returns); a revoked USAGE uses `CANNOT_RESUME_ERR`, so `ScheduleRule` does not keep resuming it. - Routine load resolves the effective group in `RoutineLoadJob.getCloudCluster()`, which backend selection, the task sent to BE, the workload group namespace, the plan context and the `ComputeGroup` column of SHOW ROUTINE LOAD all use, so every Kafka job follows it (Kinesis routine load is rejected in cloud mode, the only mode with compute groups). `RoutineLoadTaskScheduler` re-checks the declared group before backend allocation and before `beginTxn`. The partition refresh of a Kafka job runs the same check before its metadata request, which goes through the group's backends and, after RESUME, comes before any task: a dropped group pauses the job with `Compute group '<name>' not found.` instead of `no alive backends` and advice to check the Kafka properties. An auto resume attempt skips that check: it refreshes a job that is still PAUSED, which cannot be paused again with another reason, so the attempt goes on to NEED_SCHEDULE and the check before the first task records a revoked USAGE as `CANNOT_RESUME_ERR`. Every pause from the refresh (this check, a failed Kafka metadata request, a failed Kinesis shard listing) goes through `RoutineLoadJob.pauseIfNeedSchedule()`, which takes the job lock and checks NEED_SCHEDULE again, so it cannot undo a concurrent STOP. CREATE validates the workload group in the declared group's namespace. An ALTER that sets `compute_group` validates in that group the workload group the job has afterwards (the one the ALTER sets, or else the one it keeps), as the owner the tasks run as. An ALTER that sets `workload_group` alone is still not checked, as on master. The statement re-parsed at metadata load adopts both names without resolving either, because a failure there would cancel the job for good: the declared group can have been dropped (and the stored statement is the CREATE one, so a job moved to another group by ALTER still names its old group there), and on image load the workload groups are not loaded yet. - CREATE / ALTER ROUTINE LOAD start a suspended declared group the way `USE @<group>` does (`waitForAutoStart`), on the statement thread before any job lock, since they send their Kafka metadata requests (custom partitions, offset conversion) through that group's backends. An ALTER that sets `compute_group` and `kafka_offsets` together converts the offsets in the group it declares, because the job's current group may be gone. The scheduler keeps resolving backends by name without starting a suspended group, as master does for every cloud routine load job. - SHOW CREATE ROUTINE LOAD writes `compute_group` when the job declares one. It writes the declared value, which follows ALTER, and never the effective group, so a job recreated from the statement is pinned the same way, and the statement of a job that declares nothing does not pin the group it inherited. SHOW CREATE MATERIALIZED VIEW already renders every MV property. - An async MV applies the declared group, then the session's group, on the planning context of the task and on every refresh context, and re-checks the declared group before each refresh. Each refresh context reads the declaration once and binds, records and re-checks that value. `REFRESH ... INCREMENTAL WITH DRY RUN` runs outside the task, on the current FE, in a context of its own: it binds the declared group to that context and re-checks it the same way. `EXPLAIN REFRESH` binds it to its planning context without the re-check, as the task does for its own. Neither borrows the session's group for an MV that declares none, as on master. Where master differs from branch-4.x: - master's `MTMVTask` splits a refresh into the IVM incremental path and the partition / complete path, each with its own context. `applyComputeGroup()` is factored out of the existing `setupComputeGroup()`, and the re-check runs in both paths. - master's routine load also has a Kinesis source, but `KinesisRoutineLoadJob` rejects cloud mode, so a Kinesis job never has a compute group to declare. - `REFRESH ... WITH DRY RUN` (from master's IVM) and `EXPLAIN REFRESH` exist only on master. Both build their context in `RefreshMTMVCommand` instead of `MTMVTask`, so that command binds the declared group itself, and the dry run re-checks it. - SHOW CREATE ROUTINE LOAD writes the declared `compute_group`; branch-4.x does not. - An ALTER that sets `compute_group` checks the job's workload group in that group; branch-4.x does not, so there a job moved to a group without its workload group fails every task. - The Kafka partition refresh checks the declared group before its metadata request, except on an auto resume attempt; branch-4.x has no check there. - Each MV refresh context reads the declaration once to bind and re-check it; branch-4.x reads it twice. - The 4.x unit test for a task without a task context is not carried over: `MTMVJob` always creates a task with a context, and master's `setupComputeGroup()` already relies on it. - The three unit test classes are converted to JUnit 5, as master requires, and `MTMVComputeGroupTest` gains two cases for the master-only parts: `setupComputeGroup()` applies and records the declared group, and the MV re-check fails for a declared group that no longer exists while it ignores the session's group. `RefreshMTMVCommandTest` gains four for the dry run and EXPLAIN REFRESH contexts, and `RoutineLoadJobTest` one for SHOW CREATE ROUTINE LOAD. - `ConnectContextTest.testThreadInfoUsesSessionCloudCluster` (#68111) sets the session variable directly instead of calling `ConnectContext.setCloudCluster()`, which is now a no-op outside cloud mode; branch-4.x made the same change when it picked #68111 (#68948). **Results.** The unit tests below pass locally. The regression suites need a cloud cluster and were not run locally: `test_routine_load_compute_group` and `test_mtmv_compute_group` are p0 suites, so P0 runs their non-cloud rejection checks and cloud_p0 runs them in cloud mode (the routine load part also needs Kafka); `test_compute_group_binding_isolation` is a docker suite that starts its own two-group cloud cluster.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem Summary: due to #50783 ,SHOW PROCESSLIST read the cached policy-selected compute group directly, while query routing gives precedence to the session compute group. When a session or per-query hint selected another compute group, the query ran there but PROCESSLIST reported the stale cached group. Prefer the active session compute group when rendering process rows and retain the cached value as the fallback.
SHOW PROCESSLIST now reports the compute group used by the running query.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)