Skip to content

[fix](fe) Check the workload group ALTER ROUTINE LOAD sets, as CREATE does - #70130

Merged
morningman merged 2 commits into
apache:masterfrom
morningman:b2m-t44c-alter-routine-load-workload-group
Nov 5, 2026
Merged

morningman merged 2 commits into
apache:masterfrom
morningman:b2m-t44c-alter-routine-load-workload-group

Conversation

@morningman

@morningman morningman commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: #69749, #50563, #49960, #53490

Problem Summary:

In short. ALTER ROUTINE LOAD ... PROPERTIES("workload_group" = "...") accepts any name. CREATE ROUTINE LOAD checks the workload group as the job's owner in the compute group the job runs in, and every task of the job resolves it the same way, so a name that is missing there, or that the owner may not use, is accepted by the ALTER and then fails every task. This PR checks it at ALTER time, as CREATE does.

Background. A workload group belongs to a compute group in cloud mode, and to the owner's resource group outside it. Each task looks the job's workload group up as the owner in the job's compute group (KafkaTaskInfo#rePlan, KinesisTaskInfo#rePlan). Kafka jobs used to check an altered workload group (#50563, in the AlterRoutineLoadStmt path). The Nereids AlterRoutineLoadCommand path (#49960) did not carry that check, and #53490 removed the old path. #69749 brought a check back only for an ALTER that moves a Kafka job to another compute group. The comment that delays the check ("we can only get clusterId when alter job") is no longer true: the ALTER has the job at hand. Found while reviewing #69749 (thread 4415694066).

The problem, and what it cost. Measured on master 856194ece0e with unit tests; each ALTER below goes straight to the PAUSED check, so on a paused job it is accepted and written to the edit log:

Job ALTER Result
Kafka, cloud mode, runs in cg_a "workload_group" = "wg_b", which only cg_b has accepted
Kafka, outside cloud mode a workload group that does not exist accepted
Kinesis, outside cloud mode a workload group that does not exist accepted

After RESUME, the first task of each such job fails with Can not find workload group ..., and the job pauses with CREATE_TASKS_ERR. That code is retryable, so auto resume repeats this with backoff (20 s up to 300 s) for as long as the job exists. The ALTER that caused it reported success.

How this PR fixes it.

  • RoutineLoadJob#checkAlteredWorkloadGroup, called by the Kafka and the Kinesis job first thing under the job's write lock, before the PAUSED check, so that of two ALTERs of one job running at once the later one checks what the earlier one left. The Kafka offset conversion stays before the lock, so an ALTER with Kafka offsets and a workload group it cannot use converts the offsets before it fails. When an ALTER sets a workload group or declares a compute group, it checks the resulting workload group in the resulting compute group, with CreateRoutineLoadInfo.checkWorkloadGroup as CREATE does. The resulting workload group is the one the ALTER sets, else the one the job keeps; the resulting compute group is the one the ALTER declares, else the job's. The check runs as the owner, because the tasks run as the owner. A rejected ALTER changes nothing and writes no edit log. The replay of an ALTER does not go through this path.
  • The Kafka job's check for a move to another compute group ([feature](cloud) Support declaring compute_group on routine load and async MV #69749) is replaced by this one; it checks the same thing for that ALTER.
  • The comments in AlterRoutineLoadCommand that pointed at the old place, or at the old reason for not checking, are corrected.
  • test_routine_load_alter altered a job to alter_routine_load_group without creating the group: it worked only because the ALTER did not check, or because test_routine_load happened to create the group first. It now creates the group, the way test_routine_load does, and checks that an ALTER to a missing group fails.

Results.

ALTER Before After
a workload group the job's compute group has, and the owner may use accepted accepted
a workload group the job's compute group lacks, or the owner may not use accepted, then every task fails and auto resume loops rejected, e.g. Can not find workload group wg_b in compute croup cg_a; the job is unchanged
a compute group, keeping a workload group the new group lacks (Kafka) rejected (#69749) rejected
neither property not checked not checked

branch-4.1 and branch-4.2 have the same gap.

Release note

ALTER ROUTINE LOAD now rejects a workload_group that does not exist in the job's compute group (outside cloud mode, the job owner's resource group), or that the job's owner may not use, as CREATE ROUTINE LOAD does. Such an ALTER used to succeed, and then every task of the job failed.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

    RoutineLoadComputeGroupTest gets 3 cases. In cloud mode, a workload-group-only ALTER is checked in the job's compute group, both for a job running in the group snapshotted at creation and for a job that declares its group. Outside cloud mode, Kafka and Kinesis jobs are checked, and every lookup runs as the owner. A third runs two ALTERs of one paused job at once: the second starts from inside the first one's workload group lookup and waits for the job lock, then fails with Can not find workload group wg_a in compute croup cg_b, and the job keeps wg_old. The workload group mock moves into a shared helper. One case of [feature](cloud) Support declaring compute_group on routine load and async MV #69749 now asserts that an ALTER setting neither property looks nothing up, instead of asserting that a workload-group-only ALTER looks nothing up. The first two fail on master; the third fails with the check before the job lock. 11 routine load test classes, 108 cases, pass. Each part of the fix, broken on its own, fails its case: leaving the Kinesis job unchecked, checking a move in the job's old compute group, checking only an ALTER that declares a compute group, checking as whoever runs the ALTER instead of the owner, ignoring the workload group the ALTER sets, and checking before the job lock. The test_routine_load_alter change needs Kafka, so it was only parsed offline and was not run locally.

  • Behavior changed:

    • No.
    • Yes.
      ALTER ROUTINE LOAD rejects a workload group that the job could not use, which it used to accept. A workload group now has to be created, and granted to the job's owner, before a job is altered to it, as before CREATE ROUTINE LOAD.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document

… does

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69749, apache#50563, apache#49960, apache#53490

Problem Summary: ALTER ROUTINE LOAD ... PROPERTIES("workload_group" = ...)
adopted any name without a check, while CREATE ROUTINE LOAD checks the
workload group as the job's owner in the compute group the job runs in.
Every task resolves the workload group that way (KafkaTaskInfo#rePlan,
KinesisTaskInfo#rePlan), so a name missing there, or one the owner may
not use, failed every task with "Can not find workload group ..." and
paused the job with the retryable CREATE_TASKS_ERR, which auto resume
repeats for good.

Kafka jobs had this check (apache#50563, in the AlterRoutineLoadStmt path). The
Nereids AlterRoutineLoadCommand path (apache#49960) did not carry it, and
apache#53490 removed the old path. apache#69749 brought it back only for an ALTER
that moves the job to another compute group.

Measured on master 856194e with unit tests: a Kafka job running in
cg_a accepts ("workload_group" = "wg_b") although only cg_b has wg_b, and
Kafka and Kinesis jobs outside cloud mode accept a workload group that
does not exist; the ALTER only stops at the PAUSED check.

The fix:
- RoutineLoadJob#checkAlteredWorkloadGroup, which the Kafka and the
  Kinesis job call before taking the job lock, checks the workload group
  an ALTER leaves the job with whenever the ALTER sets a workload group
  or declares a compute group: the one it sets, else the one the job
  keeps, in the compute group it declares, else the job's, as the owner,
  with CreateRoutineLoadInfo.checkWorkloadGroup like CREATE. The replay
  of an ALTER does not go through it. The Kafka job's check for a move
  to another compute group (apache#69749) becomes this one.
- test_routine_load_alter creates the workload group it alters to,
  which it relied on another suite to create, and checks that an ALTER
  to a missing one fails.

branch-4.1 and branch-4.2 have the same gap.

### Release note

ALTER ROUTINE LOAD rejects a workload_group that the job's compute group
(outside cloud mode, the job owner's resource group) does not have, or
that the job's owner may not use, as CREATE ROUTINE LOAD does, instead of
accepting it and failing every task of the job.

### Check List (For Author)

- Test: Unit Test, Regression test (not run locally)
    - RoutineLoadComputeGroupTest: 2 new cases (cloud mode; Kafka and
      Kinesis outside cloud mode), one case adjusted; the new cases fail
      on master
    - 11 routine load test classes, 107 cases, pass
    - test_routine_load_alter: creates its workload group, checks the
      new error; parsed offline, needs Kafka to run
- Behavior changed: Yes. ALTER ROUTINE LOAD rejects a workload group
  the job could not use, which it used to accept.
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman

Copy link
Copy Markdown
Contributor Author

run buildall

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28065 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 72e91896bbdd8760f8bd4e488e5672406b40ce52, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17577	5093	4998	4998
q2	2124	301	242	242
q3	10298	1360	792	792
q4	4679	456	335	335
q5	7420	776	534	534
q6	182	182	159	159
q7	735	791	599	599
q8	9254	1274	1044	1044
q9	5408	4124	4114	4114
q10	6815	1281	1013	1013
q11	429	234	230	230
q12	639	390	273	273
q13	18112	3578	3024	3024
q14	274	271	240	240
q15	q16	704	706	647	647
q17	1178	745	578	578
q18	6925	6172	6108	6108
q19	1110	938	585	585
q20	368	331	219	219
q21	4910	2083	2187	2083
q22	334	291	248	248
Total cold run time: 99475 ms
Total hot run time: 28065 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5536	5537	5553	5537
q2	691	550	517	517
q3	5153	5411	4922	4922
q4	1967	2112	1535	1535
q5	5231	5111	5137	5111
q6	245	234	209	209
q7	2342	2066	1830	1830
q8	3039	2629	2678	2629
q9	8055	8006	7961	7961
q10	3710	3653	3462	3462
q11	616	425	404	404
q12	696	685	516	516
q13	3464	3804	3252	3252
q14	320	299	295	295
q15	q16	685	709	674	674
q17	8397	7073	6797	6797
q18	12979	12176	12923	12176
q19	880	755	764	755
q20	2154	2133	1947	1947
q21	5538	4509	4676	4509
q22	503	447	421	421
Total cold run time: 72201 ms
Total hot run time: 65459 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 150973 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 72e91896bbdd8760f8bd4e488e5672406b40ce52, data reload: false

query5	4296	588	438	438
query6	422	201	186	186
query7	4815	446	228	228
query8	315	166	148	148
query9	8781	3160	3150	3150
query10	462	314	250	250
query11	5914	3837	3572	3572
query12	150	92	88	88
query13	1250	447	320	320
query14	6028	3915	3630	3630
query14_1	3436	3426	3400	3400
query15	197	192	171	171
query16	935	412	385	385
query17	818	661	513	513
query18	2400	400	296	296
query19	181	150	124	124
query20	93	81	79	79
query21	210	121	105	105
query22	12966	13031	12726	12726
query23	13365	12601	12579	12579
query23_1	12496	12526	12574	12526
query24	7416	830	486	486
query24_1	488	472	473	472
query25	501	396	358	358
query26	1269	260	144	144
query27	2767	438	270	270
query28	4657	2058	2049	2049
query29	1542	573	430	430
query30	288	222	187	187
query31	980	915	804	804
query32	144	92	80	80
query33	513	285	223	223
query34	986	852	505	505
query35	728	738	632	632
query36	821	802	750	750
query37	120	96	81	81
query38	1749	1696	1685	1685
query39	783	781	737	737
query39_1	732	714	742	714
query40	212	110	98	98
query41	64	63	61	61
query42	84	84	77	77
query43	346	363	307	307
query44	1299	720	699	699
query45	174	178	164	164
query46	829	956	564	564
query47	2945	2997	2751	2751
query48	300	304	216	216
query49	562	397	284	284
query50	666	277	212	212
query51	10443	10732	10480	10480
query52	86	83	70	70
query53	179	204	152	152
query54	226	192	171	171
query55	69	78	66	66
query56	242	211	193	193
query57	1658	1608	1564	1564
query58	285	263	258	258
query59	2108	2240	1997	1997
query60	296	249	220	220
query61	161	165	168	165
query62	392	329	279	279
query63	190	156	161	156
query64	2886	1062	879	879
query65	3335	3259	3253	3253
query66	1804	428	321	321
query67	20363	20124	20030	20030
query68	2944	1018	610	610
query69	387	295	255	255
query70	851	783	771	771
query71	297	237	214	214
query72	2893	2696	2288	2288
query73	506	533	302	302
query74	4650	4510	4316	4316
query75	2335	2289	1903	1903
query76	2311	1008	587	587
query77	343	347	263	263
query78	9860	9706	9130	9130
query79	984	856	518	518
query80	1264	419	352	352
query81	579	314	262	262
query82	636	133	101	101
query83	303	206	186	186
query84	308	123	102	102
query85	867	475	373	373
query86	407	240	228	228
query87	2014	1960	1793	1793
query88	3784	2776	2746	2746
query89	326	281	243	243
query90	1887	166	160	160
query91	167	152	119	119
query92	96	89	86	86
query93	1012	982	575	575
query94	619	286	280	280
query95	577	328	319	319
query96	681	480	236	236
query97	2433	2395	2316	2316
query98	162	151	146	146
query99	775	767	644	644
Total cold run time: 235241 ms
Total hot run time: 150973 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.2 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 72e91896bbdd8760f8bd4e488e5672406b40ce52, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.05	0.05
query3	0.26	0.09	0.09
query4	1.60	0.09	0.10
query5	0.23	0.25	0.24
query6	1.16	0.68	0.67
query7	0.04	0.00	0.00
query8	0.05	0.03	0.04
query9	0.40	0.34	0.36
query10	0.58	0.55	0.57
query11	0.20	0.14	0.14
query12	0.18	0.14	0.14
query13	0.48	0.50	0.49
query14	1.02	1.01	1.00
query15	0.62	0.60	0.61
query16	0.31	0.36	0.35
query17	1.10	1.14	1.14
query18	0.20	0.19	0.18
query19	2.08	1.98	1.98
query20	0.02	0.02	0.01
query21	15.48	0.20	0.14
query22	4.95	0.06	0.06
query23	16.29	0.32	0.12
query24	3.16	0.58	0.41
query25	0.11	0.04	0.05
query26	0.98	0.22	0.16
query27	0.04	0.04	0.03
query28	2.94	0.80	0.37
query29	12.48	4.15	3.23
query30	0.28	0.13	0.11
query31	2.77	0.53	0.33
query32	3.22	0.59	0.50
query33	3.14	3.22	3.17
query34	15.50	3.99	3.34
query35	3.31	3.31	3.32
query36	0.57	0.41	0.43
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.18	0.14	0.15
query41	0.10	0.03	0.03
query42	0.03	0.04	0.04
query43	0.05	0.04	0.03
Total cold run time: 96.38 s
Total hot run time: 24.2 s

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41653239190

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41757855612

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@morningman

Copy link
Copy Markdown
Contributor Author

Review result: REQUEST_CHANGES (1 Blocker, 0 Major, 0 Minor, 0 Nit).

Reviewed commit: 72e91896bbdd8760f8bd4e488e5672406b40ce52, using GPT-6.1 Sol (xhigh).

F-01: Create the regression workload group in the job's compute/resource groups

            def validCluster = clusters[0][0]
            forComputeGroupStr = " for  $validCluster "
        }
        sql "create workload group if not exists alter_routine_load_group $forComputeGroupStr properties ( 'min_cpu_percent'='0');"

The fixture can create alter_routine_load_group in a different namespace from the one used by the routine-load job. The new validation then rejects the existing positive ALTER, so the suite stops before its remaining assertions.

Two supported configurations expose this when Kafka tests are enabled and the workload group does not already exist in the required namespace:

  1. Cloud: the user can access healthy cg_a and cg_b, but the current/default group is cg_b. SHOW CLUSTERS sorts the inventory by name and reports the active group separately. Row zero selects cg_a, while the ALTER check uses the job's effective group, cg_b. The positive ALTER fails with Can not find workload group alter_routine_load_group in compute croup cg_b.
  2. Non-cloud: an untagged root/admin owner has healthy default and extra-tag backends. Omitting FOR creates the workload group only in default, while AllBackendComputeGroup requires it in every backend tag. The extra-tag lookup rejects the positive ALTER.

At merge base 856194ece0e1e0ad45b46b96f26a3e2859412e06, Kafka checked workload groups only when ALTER also declared compute_group, so the same workload-group-only positive statement reached its assertions. The production validation change is intended; the positive test fixture needs to satisfy it. The severity refers to a broken test contract, with no claim of production data corruption.

Suggested fix: select the active cloud group, and create the non-cloud fixture in each owner-visible resource group. For example:

-        def forComputeGroupStr = ""
-        if (isCloudMode()) {
-            def clusters = sql " SHOW CLUSTERS; "
-            assertTrue(!clusters.isEmpty())
-            def validCluster = clusters[0][0]
-            forComputeGroupStr = " for  $validCluster "
+        def clusters = sql "SHOW CLUSTERS;"
+        assertTrue(!clusters.isEmpty())
+        def targetGroups = isCloudMode() ? clusters.findAll { it[1] == "TRUE" } : clusters
+        assertTrue(!targetGroups.isEmpty())
+        targetGroups.each { group ->
+            sql "create workload group if not exists alter_routine_load_group for ${group[0]} properties ( 'min_cpu_percent'='0');"
         }
-        sql "create workload group if not exists alter_routine_load_group $forComputeGroupStr properties ( 'min_cpu_percent'='0');"

Review scope: all six changed files and related invocation paths, with two review rounds. No additional production finding was identified. This is source-based review: no build or test was run, and the suggested patch has not been executed.

@morningman

Copy link
Copy Markdown
Contributor Author

On F-01: both configurations are reachable in the code, but neither is one this suite runs in, and the suggested patch would break the one it runs in outside cloud mode. I am keeping the fixture as it is.

  • Where the ALTER looks the group up. A job that declares no compute group runs in the one the session was using when it was created, and SHOW CLUSTERS marks that one is_current; row 0 is only the first by name. Outside cloud mode root has no resource tag, so the lookup goes through AllBackendComputeGroup and needs the group in every backend tag. So the fixture holds on a cluster with one compute group in cloud mode, or one backend tag outside it.
  • That is the topology of both pipelines that run this suite with Kafka: P0 has one BE, tagged default, and cloud_p0 has one compute group, cluster_name0. The nonConcurrent and external pipelines enable Kafka too, but run only their own groups, and this suite is only in p0. At 72e91896bbd the suite passed in P0 (build 1091160, 30 s) and in cloud_p0 (build 1091162, 26 s).
  • The fixture is the one test_routine_load has used since [Feature]bind workload group to compute group #49884 (lines 26-37). That suite creates a job with "workload_group" = "create_routine_load_group", and CREATE checks it the same way, as the owner in the job's compute group, so in either configuration you describe test_routine_load already fails at that CREATE on master. Hardening only this suite would leave the two fixtures inconsistent; if it is worth doing, it belongs in a change that covers the workload group fixtures together.
  • The suggested patch would turn P0 red. Outside cloud mode SHOW CLUSTERS lists default, and create workload group ... for default ... does not parse, because DEFAULT is a reserved keyword (extraneous input 'default'). The name would need backquotes.

So I would rate this a Minor about running the suite on a cluster with more than one compute group or resource group, not a Blocker, and leave it out of this PR.

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41783952504

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41795393550

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed PR 70130 at head 72e9189 against the supplied exact-head diff. Static review only: no builds or tests were run.

The production change correctly selects the effective workload group, compute group, and job owner for sequential ALTER validation, and the Kinesis path and replay path match the inspected task behavior. The accepted inline finding is the pre-lock validation race described below. The FE Core locking checkpoint is therefore not fully satisfied for concurrent ALTERs; exception handling and Nereids command normalization had no separate issue, and visible-version/Nereids rule checkpoints are not applicable. The prior general PR comment about the regression fixture was considered from the live discussion and is not duplicated as an inline finding. No user-provided review focus was supplied.

The review did not converge: the semantic and test delegated passes repeatedly failed with model-capacity errors, so their required NO_NEW_VALUABLE_FINDINGS responses could not be obtained. I independently swept all six changed files and related task, manager, validation, replay, and test paths; OA-1 is the only verified new finding. No existing inline blocking comment applies.

Review did not converge after 3 rounds. This automated review cannot pass the review check.

All-subagent convergence could not be established because the semantic and test delegated passes repeatedly failed with model-capacity errors; their NO_NEW_VALUABLE_FINDINGS responses are missing. The main agent independently reviewed every changed file and related path and delivered the verified OA-1 finding at the three-round cap.

…er the job lock

### What problem does this PR solve?

Issue Number: None

Related PR: apache#70130, apache#69749

Problem Summary: The previous commit checks the workload group an ALTER
ROUTINE LOAD leaves the job with before the Kafka and the Kinesis job
take their write lock, and RoutineLoadManager does not serialize two
ALTERs of one job. Two sessions altering the same paused job at once
could therefore each check a different combination and leave one that
neither checked: for a job in cg_a with wg_old, which cg_a and cg_b both
have, an ALTER moving it to cg_b and one setting wg_a, which only cg_a
has, both passed, and the job was left with wg_a in cg_b, which fails
every task with the retryable CREATE_TASKS_ERR. The check-before-lock
placement came from apache#69749, so on master two concurrent moves, one of
which also sets a workload group, already raced this way. It takes cloud
mode: outside it no ALTER changes the namespace the workload group is
looked up in.

Both job types now check first thing under the write lock, before the
PAUSED check, so the later of two concurrent ALTERs checks what the
earlier one left. The Kafka offset conversion stays before the lock: it
sends a metadata request through the group's backends and does not
depend on the workload group. So an ALTER with Kafka offsets and a
workload group it cannot use now converts the offsets before it fails.
The lookup is in memory (WorkloadGroupMgr, the access manager and the
compute group names), and the same locked section already writes the
edit log; no path takes the job lock while holding one of those.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - RoutineLoadComputeGroupTest#testConcurrentAltersCheckWhatTheEarlierOneLeft:
      the second ALTER starts from inside the first one's workload group
      lookup and waits for the job lock; with the check before the lock
      both were accepted and the job ended up with wg_a in cg_b, now the
      second fails with "Can not find workload group wg_a in compute croup
      cg_b" and the job keeps wg_old
    - 11 routine load test classes, 108 cases, pass
- Behavior changed: No
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@morningman

Copy link
Copy Markdown
Contributor Author

run buildall

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review completed at the exact PR head 7571ae7 after two bounded rounds. The new under-lock validation fixes the previously reported concurrent workload-group/compute-group tuple race; that existing P1 thread was independently rechecked and is not reposted. One new P1 remains: workload-group ALTER validation can dereference a null owner on legacy pre-v1.1 routine-load jobs instead of returning a controlled error. The full changed-file set, Kafka and Kinesis mutation/replay paths, effective cloud and non-cloud tuple selection, command analysis, and added tests were reviewed. No additional user focus was supplied. Review evidence is static inspection only; no builds or tests were executed under the review contract. All assigned reviewers returned NO_NEW_VALUABLE_FINDINGS in round 2, so the review converged.

Review converged after 2 round(s).

if (StringUtils.isEmpty(workloadGroup)) {
return;
}
CreateRoutineLoadInfo.checkWorkloadGroup(getUserIdentity(), workloadGroup,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Guard the legacy null owner before calling checkWorkloadGroup. RoutineLoadJob.userIdentity is documented as null for jobs created before v1.1, and an authorized ALTER can still reach this helper. Passing null into CreateRoutineLoadInfo.checkWorkloadGroup creates a context whose authorization path dereferences the identity (AccessTranslation.subjectOf / Auth.getRolesByUserWithLdap), so a workload-group-only ALTER throws an uncaught NullPointerException instead of a UserException. Reject this case with a controlled error (or apply an established legacy owner fallback) and cover it with a regression test.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The null owner is real, but only a job created before 1.1 has one, and on master such a job cannot run anyway, so this changes how an ALTER of it fails, not whether the job works. I am keeping the helper as it is.

  • The owner has been recorded since 1.1 ([fix](resource-tag) Consider resource tags when assigning tasks for broker & routine load #9492; meta version 110 is "For routine load user info"), and every CREATE records the creating session's identity (KafkaRoutineLoadJob.java:591, :605; Kinesis jobs only exist since 2026). Master does not load an image below meta version 140 (MetaContext.java:37-41), so such a job reaches master only by staying unfinished through every upgrade since 1.0, and ALTER finds only unfinished jobs.
  • On master that job already fails every task. KafkaTaskInfo#createRoutineLoadTask calls routineLoadJob.getUserIdentity().getQualifiedUser() unconditionally (KafkaTaskInfo.java:131, since [fix](load) fix multi table load plan fail after restart master Fe or leader change #53799), and for a single-table job rePlan first passes the same null owner to the same WorkloadGroupMgr.getWorkloadGroup (KafkaTaskInfo.java:204-215). The scheduler pauses the job with CREATE_TASKS_ERR (RoutineLoadTaskScheduler.java:229-236), and SHOW ROUTINE LOAD on its database throws at RoutineLoadJob.java:1763. No ALTER can give the job an owner, so the way out is STOP and CREATE, with or without this PR.
  • The NPE does not escape the statement: StmtExecutor returns it to the client as an error (StmtExecutor.java:1084-1091), the job's write lock is released in finally, nothing is merged and no edit log is written. Master already makes this call with the job's owner for an ALTER that moves a job with a workload group to another compute group ([feature](cloud) Support declaring compute_group on routine load and async MV #69749).
  • The existing fallbacks give no identity to check USAGE against: RoutineLoadManager#getAvailableBackendIds picks backends by replica tag, and ComputeGroupBindingUtil#checkComputeGroupBeforeTask skips its check. Skipping here would accept the workload group unchecked, which is what this PR stops. And a regression test cannot create such a job, since CREATE always records the owner.

So I would rate this a Minor about the error an ALTER returns on a job that cannot run, not a P1, and leave it out of this PR. If jobs without an owner are worth handling, it should be one change that covers SHOW, the tasks and ALTER together.

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28303 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7571ae75f2fcab92061bb7a012e8ff3886bec2fe, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17585	5080	5003	5003
q2	2171	309	263	263
q3	10228	1363	850	850
q4	4677	458	338	338
q5	7423	779	541	541
q6	187	188	156	156
q7	747	780	603	603
q8	9249	1264	1056	1056
q9	5459	4155	4157	4155
q10	6830	1284	1028	1028
q11	419	258	232	232
q12	639	391	272	272
q13	18114	3535	3030	3030
q14	272	275	253	253
q15	q16	716	707	676	676
q17	1182	745	568	568
q18	6986	6157	6085	6085
q19	1112	943	595	595
q20	372	328	232	232
q21	5271	2110	2229	2110
q22	337	291	257	257
Total cold run time: 99976 ms
Total hot run time: 28303 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5591	5592	5579	5579
q2	685	553	546	546
q3	5174	5372	4917	4917
q4	1967	2126	1513	1513
q5	5256	5159	5453	5159
q6	247	211	168	168
q7	2385	2057	1871	1871
q8	2977	2675	2666	2666
q9	8028	8065	7963	7963
q10	3718	3654	3576	3576
q11	626	450	425	425
q12	691	680	539	539
q13	3506	3868	3261	3261
q14	320	315	273	273
q15	q16	717	723	642	642
q17	8330	7224	6906	6906
q18	13008	12163	12912	12163
q19	895	766	768	766
q20	2161	2142	1927	1927
q21	5504	4603	4693	4603
q22	484	461	433	433
Total cold run time: 72270 ms
Total hot run time: 65896 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 151419 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7571ae75f2fcab92061bb7a012e8ff3886bec2fe, data reload: false

query5	4312	585	440	440
query6	433	211	200	200
query7	4818	449	245	245
query8	324	163	153	153
query9	8774	3219	3213	3213
query10	478	303	261	261
query11	5971	3851	3553	3553
query12	145	92	85	85
query13	1244	450	323	323
query14	6150	3959	3648	3648
query14_1	3392	3441	3419	3419
query15	211	196	189	189
query16	932	431	425	425
query17	835	637	516	516
query18	2396	404	306	306
query19	186	159	132	132
query20	84	86	82	82
query21	211	127	106	106
query22	13020	13008	12796	12796
query23	13355	12773	12359	12359
query23_1	12420	12439	12516	12439
query24	7347	825	456	456
query24_1	477	474	486	474
query25	514	390	360	360
query26	1271	253	145	145
query27	2778	435	273	273
query28	4640	2031	2030	2030
query29	1559	624	454	454
query30	294	218	176	176
query31	1013	920	804	804
query32	139	88	87	87
query33	517	291	230	230
query34	913	871	521	521
query35	736	750	666	666
query36	824	805	702	702
query37	129	93	79	79
query38	1777	1821	1735	1735
query39	806	785	755	755
query39_1	745	736	753	736
query40	220	116	93	93
query41	69	64	63	63
query42	89	86	82	82
query43	353	363	318	318
query44	1375	737	742	737
query45	186	181	165	165
query46	835	982	592	592
query47	2658	3026	2838	2838
query48	299	311	215	215
query49	573	429	312	312
query50	692	278	212	212
query51	10690	10457	10443	10443
query52	80	80	70	70
query53	184	217	154	154
query54	245	189	185	185
query55	80	69	64	64
query56	234	216	204	204
query57	1701	1627	1577	1577
query58	283	261	258	258
query59	2166	2166	1905	1905
query60	302	242	223	223
query61	149	161	172	161
query62	396	347	287	287
query63	192	163	163	163
query64	2892	1062	875	875
query65	3351	3277	3285	3277
query66	1787	438	324	324
query67	19919	20124	19979	19979
query68	2964	1026	639	639
query69	399	290	256	256
query70	832	768	780	768
query71	307	236	223	223
query72	2852	2774	2291	2291
query73	509	531	291	291
query74	4637	4519	4339	4339
query75	2330	2291	1927	1927
query76	2314	1003	609	609
query77	349	372	271	271
query78	9795	9708	9167	9167
query79	992	873	550	550
query80	766	406	334	334
query81	576	322	271	271
query82	577	136	107	107
query83	307	197	184	184
query84	314	121	103	103
query85	859	421	401	401
query86	362	236	226	226
query87	2014	1964	1797	1797
query88	3818	2785	2758	2758
query89	339	286	244	244
query90	1813	175	174	174
query91	167	168	127	127
query92	102	80	90	80
query93	1015	1031	586	586
query94	513	317	274	274
query95	568	389	316	316
query96	680	490	238	238
query97	2456	2422	2311	2311
query98	157	155	149	149
query99	765	757	645	645
Total cold run time: 234296 ms
Total hot run time: 151419 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.18 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7571ae75f2fcab92061bb7a012e8ff3886bec2fe, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.05	0.05
query3	0.26	0.09	0.09
query4	1.61	0.12	0.10
query5	0.24	0.26	0.25
query6	1.17	0.69	0.67
query7	0.04	0.01	0.01
query8	0.05	0.03	0.04
query9	0.41	0.35	0.35
query10	0.61	0.58	0.60
query11	0.20	0.15	0.13
query12	0.17	0.15	0.15
query13	0.49	0.49	0.49
query14	1.05	1.02	1.00
query15	0.62	0.60	0.61
query16	0.36	0.36	0.35
query17	1.14	1.12	1.16
query18	0.21	0.20	0.20
query19	2.15	1.95	1.99
query20	0.02	0.01	0.02
query21	15.42	0.21	0.13
query22	4.89	0.05	0.05
query23	16.27	0.31	0.12
query24	2.96	0.56	0.36
query25	0.12	0.04	0.05
query26	0.87	0.22	0.15
query27	0.04	0.04	0.04
query28	3.32	0.76	0.37
query29	12.49	4.17	3.24
query30	0.28	0.13	0.10
query31	2.76	0.56	0.33
query32	3.22	0.59	0.50
query33	3.25	3.14	3.31
query34	15.37	3.96	3.35
query35	3.35	3.34	3.35
query36	0.54	0.43	0.41
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.16	0.15
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 96.6 s
Total hot run time: 24.18 s

@liaoxin01 liaoxin01 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@924060929 924060929 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Routine-load tasks resolve workload groups using the job owner and effective compute group. This change validates the resulting configuration under the job write lock, before mutation and edit logging, for both Kafka and Kinesis.

Independent static review found no blocking P0/P1/P2 issue in the FE changes. Tuple selection matches task consumption, concurrent ALTER validation is serialized with mutation, and replay bypasses the new validation. The previous race is fixed. The legacy null-owner case changes error reporting for an already unusable job, and the multi-group fixture limitation does not establish a failure in the inspected pipeline topology; neither warrants blocking this PR.

APPROVE for the reviewed FE scope at 7571ae7. No builds or tests were run. The supplied CI snapshot reports successful P0, cloud_p0 and FE UT checks, but coverage and code-review failures remain; this review does not establish merge readiness.

Automated FE-focused static review (gpt-6.1-sol, high). No build or tests were run.

@morningman
morningman merged commit cb5b9cb into apache:master Nov 5, 2026
45 of 47 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants