Skip to content

[fix](fe) Report active compute group in processlist - #68111

Merged
deardeng merged 1 commit into
apache:masterfrom
deardeng:fix-cluster-PROCESSLIST
Sep 18, 2026
Merged

deardeng merged 1 commit into
apache:masterfrom
deardeng:fix-cluster-PROCESSLIST

Conversation

@deardeng

@deardeng deardeng commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Problem Summary: due to #50783 ,SHOW PROCESSLIST read the cached policy-selected compute group directly, while query routing gives precedence to the session compute group. When a session or per-query hint selected another compute group, the query ran there but PROCESSLIST reported the stale cached group. Prefer the active session compute group when rendering process rows and retain the cached value as the fallback.

SHOW PROCESSLIST now reports the compute group used by the running query.

  • Test: Unit Test
    • ./run-fe-ut.sh --run org.apache.doris.qe.ConnectContextTest
  • Behavior changed: Yes. The CloudCluster column now reports the active session compute group instead of a stale cached value.
  • Does this need documentation: No

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

Issue Number: None

Related PR: None

Problem Summary: SHOW PROCESSLIST read the cached policy-selected compute group directly, while query routing gives precedence to the session compute group. When a session or per-query hint selected another compute group, the query ran there but PROCESSLIST reported the stale cached group. Prefer the active session compute group when rendering process rows and retain the cached value as the fallback.

SHOW PROCESSLIST now reports the compute group used by the running query.

- Test: Unit Test
    - ./run-fe-ut.sh --run org.apache.doris.qe.ConnectContextTest
- Behavior changed: Yes. The CloudCluster column now reports the active session compute group instead of a stale cached value.
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@deardeng

Copy link
Copy Markdown
Contributor Author

run buildall

@deardeng

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review outcome: request changes for PROCESSLIST diagnostic correctness.

The direct precedence test covers only a same-thread field read. The implementation can still report a value that did not route the displayed query: it exposes the variable in non-cloud mode, and in cloud mode it reads mutable session state rather than a stable statement-scoped routing identity.

Checkpoint conclusions:

  • Goal and tests: incomplete. The ordinary direct cloud precedence works, but non-cloud, prepared/multi-statement, temporary SessionVariable replacement, and Arrow Flight lifecycles are not correct or covered.
  • Scope and compatibility: the code change is small and wire/storage compatible, but the new behavior is incorrectly broadened to shared-nothing mode. No FE-BE schema, persisted format, or symbol change was found.
  • Concurrency and lifecycle: local and all-FE/RPC PROCESSLIST share this renderer. Prepared execution lacks publication for a late SET_VAR write; view analysis safely publishes a replacement object that drops the group; Flight reverts SET_VAR while the query is still displayed during schema fetch.
  • Parallel paths and special conditions: ordinary forwarding, persistent USE/SET, single-statement COM_QUERY, and null/empty fallback were traced without another distinct defect. Post-close deferred Flight work remains the pre-existing idle-session model.
  • Configuration, persistence, transactions, and data writes: not applicable; no related change was found.
  • Performance and observability: direct cost is negligible, but the affected column is itself the observability surface and is wrong in the cases above. No additional metric is required once the snapshot is correct.
  • Test quality: the added JUnit test has the right column index but does not enable cloud mode or exercise routing, concurrency, replacement, local/RPC collection, or Flight lifecycle.
  • PR metadata: the body text says Unit Test and behavior changed, while the template boxes say Regression test and behavior unchanged; the release note also says None despite a user-visible PROCESSLIST change.

No additional user focus was supplied. Per the review contract, this was a static-only review; no build or tests were run locally.

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 27998 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9f303f40e870355d8bfd7f13b6a138f9de1a39a5, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17669	4018	4022	4018
q2	2299	339	292	292
q3	10014	1431	818	818
q4	4685	485	344	344
q5	7482	841	560	560
q6	189	180	143	143
q7	726	792	624	624
q8	9321	1692	1483	1483
q9	5440	4273	4176	4176
q10	6729	1668	1358	1358
q11	450	271	254	254
q12	632	423	315	315
q13	18105	2642	2006	2006
q14	265	264	230	230
q15	q16	725	719	673	673
q17	1850	1108	992	992
q18	6557	5602	5574	5574
q19	1349	1242	1007	1007
q20	468	397	263	263
q21	6016	2863	2548	2548
q22	467	372	320	320
Total cold run time: 101438 ms
Total hot run time: 27998 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4881	5036	4814	4814
q2	822	644	565	565
q3	4875	5181	4751	4751
q4	2321	2334	1515	1515
q5	4633	4447	4756	4447
q6	243	187	142	142
q7	1821	1754	1520	1520
q8	2534	2230	2203	2203
q9	7281	7320	7297	7297
q10	4313	4270	3845	3845
q11	545	460	378	378
q12	701	715	509	509
q13	2367	2651	2028	2028
q14	269	288	247	247
q15	q16	684	684	606	606
q17	7486	6792	6699	6699
q18	11894	11014	11773	11014
q19	1072	1053	975	975
q20	2197	2195	1937	1937
q21	5498	4552	4735	4552
q22	515	452	434	434
Total cold run time: 66952 ms
Total hot run time: 60478 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 153470 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9f303f40e870355d8bfd7f13b6a138f9de1a39a5, data reload: false

query5	4325	615	462	462
query6	450	188	169	169
query7	4831	540	291	291
query8	328	180	168	168
query9	8752	4030	4025	4025
query10	450	307	260	260
query11	5826	2243	1999	1999
query12	158	99	99	99
query13	1265	538	418	418
query14	6519	4619	4235	4235
query14_1	4060	4026	4021	4021
query15	202	189	176	176
query16	972	444	419	419
query17	890	651	512	512
query18	2456	449	328	328
query19	191	173	130	130
query20	104	107	102	102
query21	213	135	115	115
query22	13080	12981	12894	12894
query23	15576	14554	13955	13955
query23_1	14127	14107	14090	14090
query24	7575	1711	1218	1218
query24_1	1270	1186	1205	1186
query25	538	415	332	332
query26	1269	317	169	169
query27	2683	574	343	343
query28	4570	1951	1953	1951
query29	1077	626	476	476
query30	319	238	201	201
query31	900	749	632	632
query32	144	96	98	96
query33	522	324	264	264
query34	1177	1093	643	643
query35	739	770	664	664
query36	793	784	735	735
query37	154	108	89	89
query38	1836	1776	1720	1720
query39	697	701	677	677
query39_1	651	649	661	649
query40	220	126	108	108
query41	72	69	73	69
query42	98	92	95	92
query43	346	354	300	300
query44	1373	685	707	685
query45	192	179	169	169
query46	1024	1170	707	707
query47	1501	1513	1391	1391
query48	400	413	288	288
query49	618	414	300	300
query50	944	348	244	244
query51	10648	10670	10214	10214
query52	85	89	75	75
query53	238	264	184	184
query54	258	225	247	225
query55	86	75	70	70
query56	242	226	227	226
query57	1385	1450	1326	1326
query58	249	225	223	223
query59	1986	2060	1833	1833
query60	310	245	231	231
query61	164	156	164	156
query62	402	316	275	275
query63	223	178	179	178
query64	3025	1098	836	836
query65	4020	3985	3972	3972
query66	1799	426	312	312
query67	20315	20121	19985	19985
query68	3455	1494	938	938
query69	430	304	265	265
query70	965	881	890	881
query71	306	235	223	223
query72	2919	2535	2144	2144
query73	830	811	414	414
query74	4705	4492	4298	4298
query75	2298	2278	1932	1932
query76	2432	1117	683	683
query77	362	387	292	292
query78	9186	9127	8440	8440
query79	1359	1245	713	713
query80	735	452	374	374
query81	500	280	243	243
query82	616	160	122	122
query83	359	280	249	249
query84	305	143	110	110
query85	875	470	391	391
query86	379	245	218	218
query87	1998	2007	1837	1837
query88	3697	2752	2734	2734
query89	358	291	249	249
query90	1845	182	182	182
query91	165	157	120	120
query92	102	89	91	89
query93	1492	1461	809	809
query94	591	331	257	257
query95	674	354	417	354
query96	1031	799	350	350
query97	2424	2424	2344	2344
query98	199	186	184	184
query99	740	709	603	603
Total cold run time: 242352 ms
Total hot run time: 153470 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.09 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9f303f40e870355d8bfd7f13b6a138f9de1a39a5, data reload: false

query1	0.01	0.01	0.00
query2	0.10	0.05	0.05
query3	0.26	0.13	0.13
query4	1.61	0.15	0.13
query5	0.25	0.21	0.22
query6	1.15	0.98	0.95
query7	0.04	0.00	0.00
query8	0.06	0.04	0.04
query9	0.39	0.33	0.34
query10	0.55	0.54	0.54
query11	0.21	0.15	0.14
query12	0.18	0.15	0.15
query13	0.46	0.47	0.47
query14	0.96	0.95	0.92
query15	0.61	0.60	0.61
query16	0.31	0.33	0.33
query17	1.10	1.12	1.09
query18	0.22	0.20	0.20
query19	2.01	1.98	1.92
query20	0.02	0.01	0.01
query21	15.42	0.22	0.13
query22	4.76	0.05	0.05
query23	16.13	0.31	0.14
query24	2.94	0.42	0.33
query25	0.10	0.06	0.04
query26	0.72	0.20	0.15
query27	0.05	0.03	0.04
query28	3.54	0.80	0.36
query29	12.52	4.04	3.22
query30	0.28	0.15	0.16
query31	2.77	0.56	0.32
query32	3.22	0.60	0.49
query33	3.08	3.21	3.19
query34	15.59	3.96	3.25
query35	3.24	3.24	3.25
query36	0.55	0.44	0.43
query37	0.09	0.07	0.06
query38	0.06	0.04	0.04
query39	0.04	0.04	0.03
query40	0.18	0.15	0.15
query41	0.09	0.03	0.04
query42	0.03	0.02	0.03
query43	0.05	0.04	0.03
Total cold run time: 95.95 s
Total hot run time: 24.09 s

@deardeng
deardeng merged commit 102f697 into apache:master Sep 18, 2026
54 checks passed
github-actions Bot pushed a commit that referenced this pull request Sep 18, 2026
Problem Summary: due to #50783 ,SHOW
PROCESSLIST read the cached policy-selected compute group directly,
while query routing gives precedence to the session compute group. When
a session or per-query hint selected another compute group, the query
ran there but PROCESSLIST reported the stale cached group. Prefer the
active session compute group when rendering process rows and retain the
cached value as the fallback.

SHOW PROCESSLIST now reports the compute group used by the running
query.

- Test: Unit Test
    - ./run-fe-ut.sh --run org.apache.doris.qe.ConnectContextTest
- Behavior changed: Yes. The CloudCluster column now reports the active
session compute group instead of a stale cached value.
yiguolei pushed a commit that referenced this pull request Sep 19, 2026
Problem Summary: due to #50783 ,SHOW
PROCESSLIST read the cached policy-selected compute group directly,
while query routing gives precedence to the session compute group. When
a session or per-query hint selected another compute group, the query
ran there but PROCESSLIST reported the stale cached group. Prefer the
active session compute group when rendering process rows and retain the
cached value as the fallback.

SHOW PROCESSLIST now reports the compute group used by the running
query.

- Test: Unit Test
    - ./run-fe-ut.sh --run org.apache.doris.qe.ConnectContextTest
- Behavior changed: Yes. The CloudCluster column now reports the active
session compute group instead of a stale cached value.
yiguolei pushed a commit that referenced this pull request Sep 20, 2026
Problem Summary: due to #50783 ,SHOW
PROCESSLIST read the cached policy-selected compute group directly,
while query routing gives precedence to the session compute group. When
a session or per-query hint selected another compute group, the query
ran there but PROCESSLIST reported the stale cached group. Prefer the
active session compute group when rendering process rows and retain the
cached value as the fallback.

SHOW PROCESSLIST now reports the compute group used by the running
query.

- Test: Unit Test
    - ./run-fe-ut.sh --run org.apache.doris.qe.ConnectContextTest
- Behavior changed: Yes. The CloudCluster column now reports the active
session compute group instead of a stale cached value.
yiguolei pushed a commit that referenced this pull request Sep 20, 2026
Problem Summary: due to #50783 ,SHOW
PROCESSLIST read the cached policy-selected compute group directly,
while query routing gives precedence to the session compute group. When
a session or per-query hint selected another compute group, the query
ran there but PROCESSLIST reported the stale cached group. Prefer the
active session compute group when rendering process rows and retain the
cached value as the fallback.

SHOW PROCESSLIST now reports the compute group used by the running
query.

- Test: Unit Test
    - ./run-fe-ut.sh --run org.apache.doris.qe.ConnectContextTest
- Behavior changed: Yes. The CloudCluster column now reports the active
session compute group instead of a stale cached value.
yiguolei pushed a commit that referenced this pull request Oct 14, 2026
pick from #68111
Problem Summary: due to #50783 ,SHOW
PROCESSLIST read the cached policy-selected compute group directly,
while query routing gives precedence to the session compute group. When
a session or per-query hint selected another compute group, the query
ran there but PROCESSLIST reported the stale cached group. Prefer the
active session compute group when rendering process rows and retain the
cached value as the fallback.

SHOW PROCESSLIST now reports the compute group used by the running
query.

- Test: Unit Test
    - ./run-fe-ut.sh --run org.apache.doris.qe.ConnectContextTest
- Behavior changed: Yes. The CloudCluster column now reports the active
session compute group instead of a stale cached value.

(cherry picked from commit 102f697)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
yiguolei pushed a commit that referenced this pull request Oct 15, 2026
pick from #68111
Problem Summary: due to #50783 ,SHOW
PROCESSLIST read the cached policy-selected compute group directly,
while query routing gives precedence to the session compute group. When
a session or per-query hint selected another compute group, the query
ran there but PROCESSLIST reported the stale cached group. Prefer the
active session compute group when rendering process rows and retain the
cached value as the fallback.

SHOW PROCESSLIST now reports the compute group used by the running
query.

- Test: Unit Test
    - ./run-fe-ut.sh --run org.apache.doris.qe.ConnectContextTest
- Behavior changed: Yes. The CloudCluster column now reports the active
session compute group instead of a stale cached value.

(cherry picked from commit 102f697)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
morningman added a commit that referenced this pull request Nov 2, 2026
…async MV (#69749)

### What problem does this PR solve?

Issue Number: None

Related PR: #67010

Problem Summary:

**In short.** This ports branch-4.1 #67010 (also on branch-4.2) to
master: CREATE / ALTER ROUTINE LOAD and async materialized views accept
a `compute_group` property in cloud mode, so a background job can be
pinned to a compute group instead of inheriting one implicitly, and the
declared group is re-checked before every task.

**Background.** In cloud mode every query runs in a compute group. A
routine load job snapshots the compute group of the session that created
it, and an async MV has no compute group of its own: an automatic
refresh runs wherever `admin` resolves to, and a manual `REFRESH`
borrows the group of the session that triggered it.

**The problem, and what it cost.** On master nothing lets a user choose
where a background job runs:

| | master today | with this PR |
|---|---|---|
| Routine load | runs in whatever group the creating session was on;
ALTER cannot move it | `"compute_group" = "cg_etl"` on CREATE or ALTER
pins it |
| Async MV, automatic refresh | runs where `admin` resolves to | runs in
the declared group |
| Async MV, manual `REFRESH` | runs in the triggering session's group,
so the same MV refreshes in different places depending on who triggered
it | runs in the declared group; an MV that declares none keeps
borrowing the session's group |
| Async MV, `REFRESH ... WITH DRY RUN` and `EXPLAIN REFRESH` | run and
plan where `admin` resolves to | run and plan in the declared group; an
MV that declares none resolves as today |
| A pinned group is dropped, or the owner's USAGE on it is revoked | (no
pinning) | the next task fails with that reason instead of a later "no
available BE found" |
| `compute_group` in non-cloud mode, or `DEFAULT` | (no property) |
rejected at CREATE / ALTER |

Four smaller defects go with it on master: CREATE ROUTINE LOAD checks
the `workload_group` in the session's compute group namespace although
the job runs elsewhere; `RoutineLoadJob.plan()` builds the sink's plan
context from the raw snapshot field rather than the effective group; and
the statement re-parsed when a routine load job is loaded from metadata
resolves its `workload_group` again. On image load that lookup runs
before any workload group exists (`PersistMetaModules` reads
`routineLoadJobs` before `workloadGroups`, and until then the Env holds
the empty `WorkloadGroupMgr` its constructor created), and a failure
there sets the job to the final `CANCELLED`, so a job created with
`workload_group` is cancelled by the first restart that loads it from an
image (shown by a unit test with an empty manager, not reproduced on a
cluster). And when the partition refresh fails, which
`RoutineLoadJob.update()` runs before it takes the job lock, it pauses a
NEED_SCHEDULE job without that lock, so a STOP that completes in between
is turned back into PAUSED on the master while replay keeps STOPPED
(Kafka metadata failures and Kinesis shard listing alike).

**How this PR fixes it.** Same design as #67010:

- The declaration is stored in the existing property maps
(`RoutineLoadJob.jobProperties`, `MTMV.mvProperties`) and written only
when declared: no new persisted field, journal type or meta version, and
an absent key keeps today's implicit resolution, which is also how a
later binding model tells a pinned group from an inherited one.
`DEFAULT` is rejected because that model reserves it for "follow the
owner's default group"; non-cloud mode rejects the property, and
`ConnectContext.setCloudCluster()` is a no-op outside cloud mode.
- `ComputeGroupBindingUtil` holds the two checks:
`validateDeclaredComputeGroup` at CREATE / ALTER (USAGE for the current
user, then existence, the order of `USE @<group>`), and
`checkComputeGroupBeforeTask` before every task, against the identity
the task runs as. A missing group keeps `INTERNAL_ERR`, so a routine
load job is paused retryably and resumes by itself when the group comes
back (a group scaled to zero backends leaves the cluster map and
returns); a revoked USAGE uses `CANNOT_RESUME_ERR`, so `ScheduleRule`
does not keep resuming it.
- Routine load resolves the effective group in
`RoutineLoadJob.getCloudCluster()`, which backend selection, the task
sent to BE, the workload group namespace, the plan context and the
`ComputeGroup` column of SHOW ROUTINE LOAD all use, so every Kafka job
follows it (Kinesis routine load is rejected in cloud mode, the only
mode with compute groups). `RoutineLoadTaskScheduler` re-checks the
declared group before backend allocation and before `beginTxn`. The
partition refresh of a Kafka job runs the same check before its metadata
request, which goes through the group's backends and, after RESUME,
comes before any task: a dropped group pauses the job with `Compute
group '<name>' not found.` instead of `no alive backends` and advice to
check the Kafka properties. An auto resume attempt skips that check: it
refreshes a job that is still PAUSED, which cannot be paused again with
another reason, so the attempt goes on to NEED_SCHEDULE and the check
before the first task records a revoked USAGE as `CANNOT_RESUME_ERR`.
Every pause from the refresh (this check, a failed Kafka metadata
request, a failed Kinesis shard listing) goes through
`RoutineLoadJob.pauseIfNeedSchedule()`, which takes the job lock and
checks NEED_SCHEDULE again, so it cannot undo a concurrent STOP. CREATE
validates the workload group in the declared group's namespace. An ALTER
that sets `compute_group` validates in that group the workload group the
job has afterwards (the one the ALTER sets, or else the one it keeps),
as the owner the tasks run as. An ALTER that sets `workload_group` alone
is still not checked, as on master. The statement re-parsed at metadata
load adopts both names without resolving either, because a failure there
would cancel the job for good: the declared group can have been dropped
(and the stored statement is the CREATE one, so a job moved to another
group by ALTER still names its old group there), and on image load the
workload groups are not loaded yet.
- CREATE / ALTER ROUTINE LOAD start a suspended declared group the way
`USE @<group>` does (`waitForAutoStart`), on the statement thread before
any job lock, since they send their Kafka metadata requests (custom
partitions, offset conversion) through that group's backends. An ALTER
that sets `compute_group` and `kafka_offsets` together converts the
offsets in the group it declares, because the job's current group may be
gone. The scheduler keeps resolving backends by name without starting a
suspended group, as master does for every cloud routine load job.
- SHOW CREATE ROUTINE LOAD writes `compute_group` when the job declares
one. It writes the declared value, which follows ALTER, and never the
effective group, so a job recreated from the statement is pinned the
same way, and the statement of a job that declares nothing does not pin
the group it inherited. SHOW CREATE MATERIALIZED VIEW already renders
every MV property.
- An async MV applies the declared group, then the session's group, on
the planning context of the task and on every refresh context, and
re-checks the declared group before each refresh. Each refresh context
reads the declaration once and binds, records and re-checks that value.
`REFRESH ... INCREMENTAL WITH DRY RUN` runs outside the task, on the
current FE, in a context of its own: it binds the declared group to that
context and re-checks it the same way. `EXPLAIN REFRESH` binds it to its
planning context without the re-check, as the task does for its own.
Neither borrows the session's group for an MV that declares none, as on
master.

Where master differs from branch-4.x:

- master's `MTMVTask` splits a refresh into the IVM incremental path and
the partition / complete path, each with its own context.
`applyComputeGroup()` is factored out of the existing
`setupComputeGroup()`, and the re-check runs in both paths.
- master's routine load also has a Kinesis source, but
`KinesisRoutineLoadJob` rejects cloud mode, so a Kinesis job never has a
compute group to declare.
- `REFRESH ... WITH DRY RUN` (from master's IVM) and `EXPLAIN REFRESH`
exist only on master. Both build their context in `RefreshMTMVCommand`
instead of `MTMVTask`, so that command binds the declared group itself,
and the dry run re-checks it.
- SHOW CREATE ROUTINE LOAD writes the declared `compute_group`;
branch-4.x does not.
- An ALTER that sets `compute_group` checks the job's workload group in
that group; branch-4.x does not, so there a job moved to a group without
its workload group fails every task.
- The Kafka partition refresh checks the declared group before its
metadata request, except on an auto resume attempt; branch-4.x has no
check there.
- Each MV refresh context reads the declaration once to bind and
re-check it; branch-4.x reads it twice.
- The 4.x unit test for a task without a task context is not carried
over: `MTMVJob` always creates a task with a context, and master's
`setupComputeGroup()` already relies on it.
- The three unit test classes are converted to JUnit 5, as master
requires, and `MTMVComputeGroupTest` gains two cases for the master-only
parts: `setupComputeGroup()` applies and records the declared group, and
the MV re-check fails for a declared group that no longer exists while
it ignores the session's group. `RefreshMTMVCommandTest` gains four for
the dry run and EXPLAIN REFRESH contexts, and `RoutineLoadJobTest` one
for SHOW CREATE ROUTINE LOAD.
- `ConnectContextTest.testThreadInfoUsesSessionCloudCluster` (#68111)
sets the session variable directly instead of calling
`ConnectContext.setCloudCluster()`, which is now a no-op outside cloud
mode; branch-4.x made the same change when it picked #68111 (#68948).

**Results.** The unit tests below pass locally. The regression suites
need a cloud cluster and were not run locally:
`test_routine_load_compute_group` and `test_mtmv_compute_group` are p0
suites, so P0 runs their non-cloud rejection checks and cloud_p0 runs
them in cloud mode (the routine load part also needs Kafka);
`test_compute_group_binding_isolation` is a docker suite that starts its
own two-group cloud cluster.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants