Skip to content

[Chore](nereids) remove AlterRoutineLoadStmt - #53490

Merged
starocean999 merged 1 commit into
apache:masterfrom
yx-keith:remove-AlterRoutineLoadStmt
Jul 18, 2025
Merged

starocean999 merged 1 commit into
apache:masterfrom
yx-keith:remove-AlterRoutineLoadStmt

Conversation

@yx-keith

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@yx-keith

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

Copy link
Copy Markdown
TPC-H: Total hot run time: 34081 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 02ab60b00e128a5082cf84b99e3cd9d116d38150, data reload: false

------ Round 1 ----------------------------------
q1	17592	5236	5074	5074
q2	1919	277	182	182
q3	10323	1318	712	712
q4	10214	1024	522	522
q5	7538	2365	2399	2365
q6	177	160	125	125
q7	883	753	601	601
q8	9319	1313	1121	1121
q9	6753	5122	5108	5108
q10	6901	2383	1975	1975
q11	482	289	273	273
q12	343	364	227	227
q13	17748	3773	3099	3099
q14	233	247	221	221
q15	541	489	486	486
q16	441	445	371	371
q17	583	883	366	366
q18	7563	7276	7129	7129
q19	1213	954	580	580
q20	341	354	220	220
q21	4156	3211	2382	2382
q22	1017	1023	942	942
Total cold run time: 106280 ms
Total hot run time: 34081 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5168	5105	5112	5105
q2	241	317	225	225
q3	2150	2642	2294	2294
q4	1343	1770	1319	1319
q5	4225	4594	4568	4568
q6	214	175	132	132
q7	2007	1907	1874	1874
q8	2703	2619	2641	2619
q9	7314	7263	7198	7198
q10	3151	3297	2861	2861
q11	579	552	474	474
q12	676	835	678	678
q13	3589	3953	3397	3397
q14	412	309	284	284
q15	538	493	481	481
q16	444	498	452	452
q17	1186	1489	1375	1375
q18	8129	7720	7641	7641
q19	834	901	1058	901
q20	2063	2077	1932	1932
q21	5159	4687	4222	4222
q22	1041	1026	969	969
Total cold run time: 53166 ms
Total hot run time: 51001 ms

@doris-robot

Copy link
Copy Markdown
TPC-DS: Total hot run time: 186420 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 02ab60b00e128a5082cf84b99e3cd9d116d38150, data reload: false

query1	995	385	396	385
query2	6505	1640	1675	1640
query3	6745	214	221	214
query4	26623	23163	22966	22966
query5	4334	577	429	429
query6	305	213	202	202
query7	4635	504	287	287
query8	285	225	223	223
query9	8624	2583	2591	2583
query10	482	346	277	277
query11	15370	15083	14848	14848
query12	150	110	106	106
query13	1660	538	398	398
query14	8925	5833	5838	5833
query15	210	196	174	174
query16	7677	640	502	502
query17	1250	674	557	557
query18	2018	385	321	321
query19	189	174	152	152
query20	118	111	113	111
query21	204	130	104	104
query22	4154	4258	4031	4031
query23	34183	33217	33155	33155
query24	8395	2370	2376	2370
query25	524	452	404	404
query26	1221	261	142	142
query27	2756	492	339	339
query28	4350	2112	2103	2103
query29	721	546	442	442
query30	290	216	187	187
query31	955	837	773	773
query32	79	63	63	63
query33	559	352	337	337
query34	795	833	517	517
query35	784	819	731	731
query36	981	971	888	888
query37	107	98	73	73
query38	4147	4081	3959	3959
query39	1512	1431	1449	1431
query40	221	117	105	105
query41	58	60	52	52
query42	116	102	109	102
query43	504	513	487	487
query44	1314	820	829	820
query45	175	164	160	160
query46	829	999	629	629
query47	1785	1794	1729	1729
query48	384	425	318	318
query49	736	485	400	400
query50	622	709	398	398
query51	5507	5415	5599	5415
query52	107	107	99	99
query53	227	258	178	178
query54	561	570	502	502
query55	83	85	82	82
query56	322	305	309	305
query57	1208	1189	1137	1137
query58	261	260	272	260
query59	2560	2620	2465	2465
query60	334	318	301	301
query61	130	126	145	126
query62	808	723	639	639
query63	224	196	190	190
query64	4264	992	644	644
query65	4260	4168	4196	4168
query66	1185	410	316	316
query67	15784	15824	15557	15557
query68	7856	885	513	513
query69	469	302	275	275
query70	1248	1100	1163	1100
query71	425	331	307	307
query72	5582	4644	4600	4600
query73	612	568	352	352
query74	9179	9264	8935	8935
query75	3498	3212	2697	2697
query76	3448	1125	729	729
query77	754	380	295	295
query78	9995	10165	9389	9389
query79	2062	830	570	570
query80	569	579	446	446
query81	465	261	222	222
query82	421	133	105	105
query83	243	253	238	238
query84	249	102	84	84
query85	791	360	323	323
query86	335	315	298	298
query87	4365	4474	4454	4454
query88	3757	2332	2299	2299
query89	388	318	291	291
query90	1952	218	222	218
query91	142	149	114	114
query92	83	62	57	57
query93	1515	944	578	578
query94	715	422	307	307
query95	377	303	286	286
query96	486	584	285	285
query97	2670	2870	2682	2682
query98	224	223	208	208
query99	1349	1392	1322	1322
Total cold run time: 274610 ms
Total hot run time: 186420 ms

@doris-robot

Copy link
Copy Markdown
ClickBench: Total hot run time: 32.47 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 02ab60b00e128a5082cf84b99e3cd9d116d38150, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.24	0.07	0.07
query4	1.64	0.11	0.11
query5	0.41	0.42	0.40
query6	1.20	0.66	0.68
query7	0.03	0.02	0.01
query8	0.05	0.03	0.04
query9	0.58	0.52	0.51
query10	0.58	0.58	0.57
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.63	0.62	0.61
query14	0.81	0.84	0.84
query15	0.90	0.90	0.90
query16	0.38	0.40	0.39
query17	1.10	1.03	1.03
query18	0.23	0.21	0.21
query19	1.97	1.88	1.82
query20	0.01	0.02	0.02
query21	15.39	0.90	0.54
query22	0.76	1.19	0.72
query23	14.92	1.40	0.65
query24	6.59	2.40	0.54
query25	0.51	0.25	0.20
query26	0.67	0.17	0.16
query27	0.07	0.05	0.06
query28	9.37	0.93	0.44
query29	12.56	4.00	3.28
query30	3.10	2.96	2.94
query31	2.83	0.60	0.39
query32	3.23	0.56	0.49
query33	3.04	3.19	3.19
query34	16.11	5.37	4.75
query35	4.82	4.86	4.83
query36	0.72	0.51	0.48
query37	0.08	0.07	0.07
query38	0.06	0.04	0.04
query39	0.03	0.03	0.02
query40	0.18	0.14	0.14
query41	0.07	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.36 s
Total hot run time: 32.47 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Jul 18, 2025
@starocean999
starocean999 merged commit 9841102 into apache:master Jul 18, 2025
@yx-keith
yx-keith deleted the remove-AlterRoutineLoadStmt branch June 9, 2026 07:13
morningman added a commit that referenced this pull request Nov 5, 2026
… does (#70130)

### What problem does this PR solve?

Issue Number: None

Related PR: #69749, #50563, #49960, #53490

Problem Summary:

**In short.** `ALTER ROUTINE LOAD ... PROPERTIES("workload_group" =
"...")` accepts any name. `CREATE ROUTINE LOAD` checks the workload
group as the job's owner in the compute group the job runs in, and every
task of the job resolves it the same way, so a name that is missing
there, or that the owner may not use, is accepted by the ALTER and then
fails every task. This PR checks it at ALTER time, as CREATE does.

**Background.** A workload group belongs to a compute group in cloud
mode, and to the owner's resource group outside it. Each task looks the
job's workload group up as the owner in the job's compute group
(`KafkaTaskInfo#rePlan`, `KinesisTaskInfo#rePlan`). Kafka jobs used to
check an altered workload group (#50563, in the `AlterRoutineLoadStmt`
path). The Nereids `AlterRoutineLoadCommand` path (#49960) did not carry
that check, and #53490 removed the old path. #69749 brought a check back
only for an ALTER that moves a Kafka job to another compute group. The
comment that delays the check ("we can only get clusterId when alter
job") is no longer true: the ALTER has the job at hand. Found while
reviewing #69749 (thread 4415694066).

**The problem, and what it cost.** Measured on master `856194ece0e` with
unit tests; each ALTER below goes straight to the PAUSED check, so on a
paused job it is accepted and written to the edit log:

| Job | ALTER | Result |
|---|---|---|
| Kafka, cloud mode, runs in `cg_a` | `"workload_group" = "wg_b"`, which
only `cg_b` has | accepted |
| Kafka, outside cloud mode | a workload group that does not exist |
accepted |
| Kinesis, outside cloud mode | a workload group that does not exist |
accepted |

After RESUME, the first task of each such job fails with `Can not find
workload group ...`, and the job pauses with `CREATE_TASKS_ERR`. That
code is retryable, so auto resume repeats this with backoff (20 s up to
300 s) for as long as the job exists. The ALTER that caused it reported
success.

**How this PR fixes it.**

- `RoutineLoadJob#checkAlteredWorkloadGroup`, called by the Kafka and
the Kinesis job first thing under the job's write lock, before the
PAUSED check, so that of two ALTERs of one job running at once the later
one checks what the earlier one left. The Kafka offset conversion stays
before the lock, so an ALTER with Kafka offsets and a workload group it
cannot use converts the offsets before it fails. When an ALTER sets a
workload group or declares a compute group, it checks the resulting
workload group in the resulting compute group, with
`CreateRoutineLoadInfo.checkWorkloadGroup` as CREATE does. The resulting
workload group is the one the ALTER sets, else the one the job keeps;
the resulting compute group is the one the ALTER declares, else the
job's. The check runs as the owner, because the tasks run as the owner.
A rejected ALTER changes nothing and writes no edit log. The replay of
an ALTER does not go through this path.
- The Kafka job's check for a move to another compute group (#69749) is
replaced by this one; it checks the same thing for that ALTER.
- The comments in `AlterRoutineLoadCommand` that pointed at the old
place, or at the old reason for not checking, are corrected.
- `test_routine_load_alter` altered a job to `alter_routine_load_group`
without creating the group: it worked only because the ALTER did not
check, or because `test_routine_load` happened to create the group
first. It now creates the group, the way `test_routine_load` does, and
checks that an ALTER to a missing group fails.

**Results.**

| ALTER | Before | After |
|---|---|---|
| a workload group the job's compute group has, and the owner may use |
accepted | accepted |
| a workload group the job's compute group lacks, or the owner may not
use | accepted, then every task fails and auto resume loops | rejected,
e.g. `Can not find workload group wg_b in compute croup cg_a`; the job
is unchanged |
| a compute group, keeping a workload group the new group lacks (Kafka)
| rejected (#69749) | rejected |
| neither property | not checked | not checked |

branch-4.1 and branch-4.2 have the same gap.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants