Repository navigation
[Chore](nereids) remove AlterRoutineLoadStmt - #53490
Merged
Merged
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 34081 ms |
TPC-DS: Total hot run time: 186420 ms |
ClickBench: Total hot run time: 32.47 s |
Contributor
FE UT Coverage ReportIncrement line coverage `` 🎉 |
zddr
approved these changes
Jul 18, 2025
Contributor
|
PR approved by anyone and no changes requested. |
starocean999
approved these changes
Jul 18, 2025
Contributor
|
PR approved by at least one committer and no changes requested. |
morningman
added a commit
that referenced
this pull request
Nov 5, 2026
… does (#70130) ### What problem does this PR solve? Issue Number: None Related PR: #69749, #50563, #49960, #53490 Problem Summary: **In short.** `ALTER ROUTINE LOAD ... PROPERTIES("workload_group" = "...")` accepts any name. `CREATE ROUTINE LOAD` checks the workload group as the job's owner in the compute group the job runs in, and every task of the job resolves it the same way, so a name that is missing there, or that the owner may not use, is accepted by the ALTER and then fails every task. This PR checks it at ALTER time, as CREATE does. **Background.** A workload group belongs to a compute group in cloud mode, and to the owner's resource group outside it. Each task looks the job's workload group up as the owner in the job's compute group (`KafkaTaskInfo#rePlan`, `KinesisTaskInfo#rePlan`). Kafka jobs used to check an altered workload group (#50563, in the `AlterRoutineLoadStmt` path). The Nereids `AlterRoutineLoadCommand` path (#49960) did not carry that check, and #53490 removed the old path. #69749 brought a check back only for an ALTER that moves a Kafka job to another compute group. The comment that delays the check ("we can only get clusterId when alter job") is no longer true: the ALTER has the job at hand. Found while reviewing #69749 (thread 4415694066). **The problem, and what it cost.** Measured on master `856194ece0e` with unit tests; each ALTER below goes straight to the PAUSED check, so on a paused job it is accepted and written to the edit log: | Job | ALTER | Result | |---|---|---| | Kafka, cloud mode, runs in `cg_a` | `"workload_group" = "wg_b"`, which only `cg_b` has | accepted | | Kafka, outside cloud mode | a workload group that does not exist | accepted | | Kinesis, outside cloud mode | a workload group that does not exist | accepted | After RESUME, the first task of each such job fails with `Can not find workload group ...`, and the job pauses with `CREATE_TASKS_ERR`. That code is retryable, so auto resume repeats this with backoff (20 s up to 300 s) for as long as the job exists. The ALTER that caused it reported success. **How this PR fixes it.** - `RoutineLoadJob#checkAlteredWorkloadGroup`, called by the Kafka and the Kinesis job first thing under the job's write lock, before the PAUSED check, so that of two ALTERs of one job running at once the later one checks what the earlier one left. The Kafka offset conversion stays before the lock, so an ALTER with Kafka offsets and a workload group it cannot use converts the offsets before it fails. When an ALTER sets a workload group or declares a compute group, it checks the resulting workload group in the resulting compute group, with `CreateRoutineLoadInfo.checkWorkloadGroup` as CREATE does. The resulting workload group is the one the ALTER sets, else the one the job keeps; the resulting compute group is the one the ALTER declares, else the job's. The check runs as the owner, because the tasks run as the owner. A rejected ALTER changes nothing and writes no edit log. The replay of an ALTER does not go through this path. - The Kafka job's check for a move to another compute group (#69749) is replaced by this one; it checks the same thing for that ALTER. - The comments in `AlterRoutineLoadCommand` that pointed at the old place, or at the old reason for not checking, are corrected. - `test_routine_load_alter` altered a job to `alter_routine_load_group` without creating the group: it worked only because the ALTER did not check, or because `test_routine_load` happened to create the group first. It now creates the group, the way `test_routine_load` does, and checks that an ALTER to a missing group fails. **Results.** | ALTER | Before | After | |---|---|---| | a workload group the job's compute group has, and the owner may use | accepted | accepted | | a workload group the job's compute group lacks, or the owner may not use | accepted, then every task fails and auto resume loops | rejected, e.g. `Can not find workload group wg_b in compute croup cg_a`; the job is unchanged | | a compute group, keeping a workload group the new group lacks (Kafka) | rejected (#69749) | rejected | | neither property | not checked | not checked | branch-4.1 and branch-4.2 have the same gap.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)