Skip to content

[fix](fe) Let MV dry runs and EXPLAIN REFRESH borrow the session's compute group - #70125

Merged
morningman merged 1 commit into
apache:masterfrom
morningman:b2m-t44b-dry-run-session-compute-group
Nov 5, 2026
Merged

morningman merged 1 commit into
apache:masterfrom
morningman:b2m-t44b-dry-run-session-compute-group

Conversation

@morningman

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: #69749, #62606

Problem Summary:

In short. In cloud mode, REFRESH MATERIALIZED VIEW ... INCREMENTAL WITH DRY RUN and EXPLAIN REFRESH MATERIALIZED VIEW of an MV that declares no compute_group plan (and the dry run executes) in whatever compute group the admin user resolves to, not in the group of the session that issues them, where a manual REFRESH of the same MV runs. This PR binds both to the session's group, the way a manual REFRESH borrows it.

Background. Both statements build an internal context with the admin identity and the MV's session variables (MTMVPlanUtil.createMTMVContext). Its cloud_cluster is empty, so it resolves a group the way admin would: admin's default_compute_group, else the first group admin may use. A manual REFRESH passes the session's group to its task instead (MTMVJobManager#refreshMTMV), and the task binds it when the MV declares none (MTMVTask#applyComputeGroup). #69749 made both statements honour a declared group; an MV that declares none, which is every MV created before #69749 and any MV created without the property, still goes to admin's group. Found while reviewing #69749 (thread 4401555648).

The problem, and what it cost. Measured on master 856194ece0e with a unit test in cloud mode: groups cg_of_admin and cg_of_session, the session on USE @cg_of_session, an MV that declares no group.

manual REFRESH dry run EXPLAIN REFRESH
compute group cg_of_session cg_of_admin cg_of_admin
  • If the MV's workload_group exists only in the session's group (a workload group belongs to one compute group in cloud mode), the dry run fails with Can not find workload group <wg> in compute croup cg_of_admin, while the manual REFRESH succeeds.
  • Otherwise the dry run's delta query runs on the backends of another group, which the session's user may have no USAGE on, and starts that group if it is suspended. EXPLAIN REFRESH assigns backends there and can start it as well. Results are not affected: the dry run only reads, and storage is shared.

How this PR fixes it.

  • The private lookup of the session's group in MTMVJobManager moves unchanged to MTMVPlanUtil.getSessionComputeGroup(), so a manual REFRESH, a dry run and EXPLAIN REFRESH resolve it with the same code.
  • RefreshMTMVCommand builds both internal contexts through one helper. It reads the session's group first, while the session is still the thread-local context, then binds the declared group, else the session's group, as MTMVTask#applyComputeGroup does. With neither, the context stays unbound and the existing resolution applies; outside cloud mode nothing changes.
  • As in MTMVTask, the dry run re-checks only a declared group, since the MV never chose the session's group. EXPLAIN REFRESH executes nothing and still checks nothing.

Results.

MV session dry run / EXPLAIN REFRESH, before after
declares no group USE @cg_of_session cg_of_admin cg_of_session, as a manual REFRESH
declares no group resolves no group admin's resolution unchanged
declares cg_declared USE @cg_of_session cg_declared unchanged
any, outside cloud mode no compute group unchanged

branch-4.1 and branch-4.2 have neither statement.

Release note

In cloud mode, REFRESH MATERIALIZED VIEW ... WITH DRY RUN and EXPLAIN REFRESH MATERIALIZED VIEW of a materialized view that declares no compute_group run in the compute group of the current session, as a manual REFRESH does, instead of the group the admin user resolves to.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

    RefreshMTMVCommandTest (mocked cloud mode) gets 4 cases: without a declaration, the dry run and the EXPLAIN contexts resolve to the session's group; a declared group still wins over it; the dry run does not re-check a borrowed group; and an existing case now runs from a session that resolves no group, where both contexts stay unbound. On master the first two fail (expected: <cg_of_session> but was: <cg_of_admin>). Together with the classes that call the changed code (ExplainRefreshMTMVCommandTest, MTMVJobManagerTest, MTMVPlanUtilTest, MTMVTaskTest, MTMVComputeGroupTest, IvmIncrRefreshMTMVTest, IvmBaselineRebuildTest, ProtocolCapabilityWiringTest, NereidsParserTest), 10 classes and 335 cases pass. Each part of the fix, broken on its own, fails its case: reading the session's group after the internal context is installed, letting the session's group win over a declared one, re-checking the borrowed group, and failing when the session resolves no group. Not run on a cloud cluster: the dry run needs IVM and changelog support in the backends, so a local cloud cluster needs a master backend image; group selection and the workload group lookup both happen in the FE, which the cases exercise with the real code.

  • Behavior changed:

    • No.
    • Yes.
      In cloud mode, the dry run and EXPLAIN REFRESH of an MV that declares no compute group use the session's compute group instead of the one admin resolves to.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document

…mpute group

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69749, apache#62606

Problem Summary: In cloud mode, REFRESH MATERIALIZED VIEW ... INCREMENTAL
WITH DRY RUN and EXPLAIN REFRESH MATERIALIZED VIEW of an MV that declares
no compute_group planned (and the dry run executed) in the compute group
the admin user resolves to, not in the group of the issuing session, where
a manual REFRESH of the same MV runs.

Both statements build an internal admin context with an empty
cloud_cluster (MTMVPlanUtil.createMTMVContext), so it resolves admin's
default compute group, else the first group admin may use. A manual
REFRESH instead hands the session's group to its task
(MTMVJobManager#refreshMTMV), which binds it when the MV declares none
(MTMVTask#applyComputeGroup). apache#69749 made both statements honour a
declared group; an MV that declares none still went to admin's group.

Measured on master 856194e with a unit test in cloud mode (groups
cg_of_admin and cg_of_session, the session on USE @cg_of_session, an MV
that declares no group): the manual REFRESH runs in cg_of_session, the dry
run and EXPLAIN REFRESH resolve to cg_of_admin. When the MV's
workload_group exists only in the session's group, the dry run fails with
"Can not find workload group ... in compute croup cg_of_admin" while the
manual REFRESH succeeds; otherwise the delta runs on another group's
backends, which the user may have no USAGE on, and can start that group
if it is suspended. Results are not affected.

The fix:
- Move the private lookup of the session's group from MTMVJobManager,
  unchanged, to MTMVPlanUtil.getSessionComputeGroup(), so a manual
  REFRESH, a dry run and EXPLAIN REFRESH resolve it with the same code.
- RefreshMTMVCommand builds both internal contexts through one helper
  that reads the session's group while the session is still the
  thread-local context, then binds the declared group, else the session's
  group, as MTMVTask#applyComputeGroup does. With neither, the context
  stays unbound and the existing resolution applies.
- As in MTMVTask, the dry run re-checks only a declared group; EXPLAIN
  REFRESH checks nothing.

branch-4.1 and branch-4.2 have neither statement.

### Release note

In cloud mode, REFRESH MATERIALIZED VIEW ... WITH DRY RUN and EXPLAIN
REFRESH MATERIALIZED VIEW of a materialized view that declares no
compute_group run in the compute group of the current session, as a
manual REFRESH does, instead of the group the admin user resolves to.

### Check List (For Author)

- Test: Unit Test
    - RefreshMTMVCommandTest: 4 new cases in a mocked cloud mode, one
      case adjusted; the two cases on the session's group fail on master
    - 10 related test classes, 335 cases, pass
- Behavior changed: Yes. In cloud mode, the dry run and EXPLAIN REFRESH
  of an MV that declares no compute group use the session's compute
  group instead of the one admin resolves to.
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman

Copy link
Copy Markdown
Contributor Author

run buildall

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28635 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fe9775848fc221801a61c5ad7ab160ed6521691c, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17575	5120	5057	5057
q2	2156	320	265	265
q3	10200	1384	813	813
q4	4677	457	334	334
q5	7419	900	531	531
q6	183	182	151	151
q7	753	780	584	584
q8	9244	1332	1025	1025
q9	5487	4195	4168	4168
q10	6812	1287	1030	1030
q11	423	245	227	227
q12	640	399	277	277
q13	18143	3606	3066	3066
q14	271	277	253	253
q15	q16	723	712	661	661
q17	1204	753	580	580
q18	6936	6201	6098	6098
q19	1249	946	610	610
q20	386	330	227	227
q21	6107	2450	2403	2403
q22	395	338	275	275
Total cold run time: 100983 ms
Total hot run time: 28635 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5946	5938	6111	5938
q2	747	625	562	562
q3	5657	5743	5277	5277
q4	1993	2124	1551	1551
q5	5690	5894	5586	5586
q6	240	191	147	147
q7	2213	2077	1789	1789
q8	3006	2625	2666	2625
q9	7897	7480	7472	7472
q10	3629	3557	3134	3134
q11	580	417	397	397
q12	655	713	506	506
q13	3312	3622	3036	3036
q14	279	288	272	272
q15	q16	654	672	613	613
q17	7665	7027	6942	6942
q18	12940	12192	12879	12192
q19	925	795	859	795
q20	2140	2126	1921	1921
q21	5541	4617	4634	4617
q22	510	458	437	437
Total cold run time: 72219 ms
Total hot run time: 65809 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 150947 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fe9775848fc221801a61c5ad7ab160ed6521691c, data reload: false

query5	4336	572	448	448
query6	432	207	188	188
query7	4796	452	226	226
query8	319	159	151	151
query9	8816	3164	3170	3164
query10	493	307	251	251
query11	5937	3836	3560	3560
query12	141	90	88	88
query13	1274	439	324	324
query14	6122	3907	3673	3673
query14_1	3387	3393	3427	3393
query15	208	202	179	179
query16	946	448	426	426
query17	875	669	538	538
query18	2425	437	306	306
query19	195	166	137	137
query20	87	83	82	82
query21	212	127	103	103
query22	12911	12939	12832	12832
query23	13367	12878	12444	12444
query23_1	12556	12512	12517	12512
query24	7358	825	470	470
query24_1	489	484	495	484
query25	532	414	360	360
query26	1270	261	140	140
query27	2762	473	271	271
query28	4612	2029	2030	2029
query29	1529	570	428	428
query30	300	228	187	187
query31	1014	926	805	805
query32	138	83	84	83
query33	514	296	219	219
query34	902	838	511	511
query35	746	740	658	658
query36	810	814	753	753
query37	124	95	81	81
query38	1765	1728	1645	1645
query39	805	782	743	743
query39_1	728	725	731	725
query40	208	126	91	91
query41	68	63	61	61
query42	83	84	79	79
query43	342	356	305	305
query44	1311	692	703	692
query45	182	172	167	167
query46	912	963	571	571
query47	3034	2761	2865	2761
query48	296	303	215	215
query49	570	384	285	285
query50	667	288	209	209
query51	10328	10523	10479	10479
query52	76	78	68	68
query53	191	202	162	162
query54	244	194	179	179
query55	69	62	62	62
query56	224	211	204	204
query57	1658	1619	1560	1560
query58	291	269	265	265
query59	2238	2224	2005	2005
query60	291	249	224	224
query61	147	148	146	146
query62	390	326	280	280
query63	183	153	154	153
query64	2759	938	769	769
query65	3306	3241	3241	3241
query66	1751	419	298	298
query67	20350	20274	19939	19939
query68	3048	987	613	613
query69	401	285	256	256
query70	833	777	766	766
query71	291	219	219	219
query72	2824	2604	2261	2261
query73	508	522	294	294
query74	4603	4496	4313	4313
query75	2329	2264	1909	1909
query76	2323	995	579	579
query77	330	376	266	266
query78	9641	9732	9187	9187
query79	903	840	527	527
query80	479	436	364	364
query81	526	323	277	277
query82	202	132	108	108
query83	194	198	178	178
query84	283	129	99	99
query85	774	413	368	368
query86	276	237	223	223
query87	1961	1967	1798	1798
query88	3678	2751	2741	2741
query89	312	279	240	240
query90	2088	163	164	163
query91	159	161	127	127
query92	98	78	86	78
query93	981	991	578	578
query94	446	302	281	281
query95	563	395	307	307
query96	677	495	236	236
query97	2456	2466	2343	2343
query98	158	150	145	145
query99	757	768	660	660
Total cold run time: 233288 ms
Total hot run time: 150947 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.08 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fe9775848fc221801a61c5ad7ab160ed6521691c, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.05	0.05
query3	0.26	0.09	0.08
query4	1.61	0.10	0.08
query5	0.24	0.24	0.25
query6	1.17	0.70	0.68
query7	0.04	0.01	0.00
query8	0.05	0.04	0.04
query9	0.41	0.34	0.37
query10	0.57	0.58	0.57
query11	0.19	0.15	0.14
query12	0.18	0.16	0.14
query13	0.49	0.48	0.49
query14	1.02	1.01	0.99
query15	0.63	0.60	0.60
query16	0.35	0.35	0.33
query17	1.11	1.18	1.11
query18	0.20	0.19	0.20
query19	2.13	1.99	1.90
query20	0.02	0.02	0.01
query21	15.42	0.19	0.14
query22	4.91	0.05	0.06
query23	16.29	0.32	0.12
query24	3.10	0.45	0.41
query25	0.13	0.06	0.06
query26	1.04	0.21	0.16
query27	0.04	0.06	0.03
query28	3.38	0.80	0.34
query29	12.46	4.14	3.23
query30	0.28	0.15	0.10
query31	2.76	0.55	0.32
query32	3.22	0.59	0.50
query33	3.16	3.15	3.29
query34	15.33	3.98	3.37
query35	3.31	3.34	3.34
query36	0.56	0.48	0.42
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.04	0.03	0.03
query40	0.19	0.14	0.14
query41	0.09	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.04	0.04
Total cold run time: 96.71 s
Total hot run time: 24.08 s

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41645033727

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41654037152

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41757850781

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41777717735

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41795404454

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@morningman

Copy link
Copy Markdown
Contributor Author

run cloud_p0

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41847059461

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed PR #70125 at exact head fe97758 against base 856194e. No verified inline findings. The change captures the caller's resolved session compute group before installing an internal MTMV context, preserves declared-MV precedence, and keeps manual refresh, dry-run, and EXPLAIN REFRESH aligned. Declared-group revalidation and thread-local cleanup remain consistent with MTMVTask and ExplainCommand. Critical checkpoints: the task goal is met by the inspected control flow and focused unit coverage; the change is small and focused; no new shared concurrency or lock path is introduced; context lifecycle and failure cleanup were traced; no configuration, protocol, persistence, transaction, or data-write format change is involved; tests cover declared, session, absent, dropped, and precedence cases, with existing manual-capture coverage. There were no existing inline threads and no additional user focus. This was a static review only; no builds or tests were executed. Review converged in round 1.

Review converged after 1 round(s).

@yujun777 yujun777 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@924060929 924060929 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In cloud mode, manual MV refresh already prefers the MV's declared compute group and otherwise borrows the session's group. This change applies the same selection to DRY RUN and EXPLAIN REFRESH before installing their internal admin context.

Independent static review of fe97758 against 856194e found no blocking issues in the affected FE paths. I traced context installation/restoration, declared-group validation, backend selection, workload-group lookup, and focused unit coverage. No FE/BE protocol or persistence changes are involved. Recommend APPROVE.

No builds or tests were run. The supplied current-head snapshot reports successful P0, cloud_p0, vault_p0, FE UT, and compilation checks, but FE coverage failed. Required-check rules were not supplied; this recommendation does not establish merge readiness.

Automated FE-focused static review (gpt-6.1-sol, high). No build or tests were run.

@morningman
morningman merged commit 35209ae into apache:master Nov 5, 2026
39 of 40 checks passed
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Nov 5, 2026
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. backport/4.2-master incremental-computation reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants