Repository navigation
[fix](fe) Check the workload group ALTER ROUTINE LOAD sets, as CREATE does - #70130
morningman merged 2 commits into
Conversation
… does ### What problem does this PR solve? Issue Number: None Related PR: apache#69749, apache#50563, apache#49960, apache#53490 Problem Summary: ALTER ROUTINE LOAD ... PROPERTIES("workload_group" = ...) adopted any name without a check, while CREATE ROUTINE LOAD checks the workload group as the job's owner in the compute group the job runs in. Every task resolves the workload group that way (KafkaTaskInfo#rePlan, KinesisTaskInfo#rePlan), so a name missing there, or one the owner may not use, failed every task with "Can not find workload group ..." and paused the job with the retryable CREATE_TASKS_ERR, which auto resume repeats for good. Kafka jobs had this check (apache#50563, in the AlterRoutineLoadStmt path). The Nereids AlterRoutineLoadCommand path (apache#49960) did not carry it, and apache#53490 removed the old path. apache#69749 brought it back only for an ALTER that moves the job to another compute group. Measured on master 856194e with unit tests: a Kafka job running in cg_a accepts ("workload_group" = "wg_b") although only cg_b has wg_b, and Kafka and Kinesis jobs outside cloud mode accept a workload group that does not exist; the ALTER only stops at the PAUSED check. The fix: - RoutineLoadJob#checkAlteredWorkloadGroup, which the Kafka and the Kinesis job call before taking the job lock, checks the workload group an ALTER leaves the job with whenever the ALTER sets a workload group or declares a compute group: the one it sets, else the one the job keeps, in the compute group it declares, else the job's, as the owner, with CreateRoutineLoadInfo.checkWorkloadGroup like CREATE. The replay of an ALTER does not go through it. The Kafka job's check for a move to another compute group (apache#69749) becomes this one. - test_routine_load_alter creates the workload group it alters to, which it relied on another suite to create, and checks that an ALTER to a missing one fails. branch-4.1 and branch-4.2 have the same gap. ### Release note ALTER ROUTINE LOAD rejects a workload_group that the job's compute group (outside cloud mode, the job owner's resource group) does not have, or that the job's owner may not use, as CREATE ROUTINE LOAD does, instead of accepting it and failing every task of the job. ### Check List (For Author) - Test: Unit Test, Regression test (not run locally) - RoutineLoadComputeGroupTest: 2 new cases (cloud mode; Kafka and Kinesis outside cloud mode), one case adjusted; the new cases fail on master - 11 routine load test classes, 107 cases, pass - test_routine_load_alter: creates its workload group, checks the new error; parsed offline, needs Kafka to run - Behavior changed: Yes. ALTER ROUTINE LOAD rejects a workload group the job could not use, which it used to accept. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
TPC-H: Total hot run time: 28065 ms |
TPC-DS: Total hot run time: 150973 ms |
ClickBench: Total hot run time: 24.2 s |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
/review |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
Review result: REQUEST_CHANGES (1 Blocker, 0 Major, 0 Minor, 0 Nit). Reviewed commit: F-01: Create the regression workload group in the job's compute/resource groups
def validCluster = clusters[0][0]
forComputeGroupStr = " for $validCluster "
}
sql "create workload group if not exists alter_routine_load_group $forComputeGroupStr properties ( 'min_cpu_percent'='0');"The fixture can create Two supported configurations expose this when Kafka tests are enabled and the workload group does not already exist in the required namespace:
At merge base Suggested fix: select the active cloud group, and create the non-cloud fixture in each owner-visible resource group. For example: - def forComputeGroupStr = ""
- if (isCloudMode()) {
- def clusters = sql " SHOW CLUSTERS; "
- assertTrue(!clusters.isEmpty())
- def validCluster = clusters[0][0]
- forComputeGroupStr = " for $validCluster "
+ def clusters = sql "SHOW CLUSTERS;"
+ assertTrue(!clusters.isEmpty())
+ def targetGroups = isCloudMode() ? clusters.findAll { it[1] == "TRUE" } : clusters
+ assertTrue(!targetGroups.isEmpty())
+ targetGroups.each { group ->
+ sql "create workload group if not exists alter_routine_load_group for ${group[0]} properties ( 'min_cpu_percent'='0');"
}
- sql "create workload group if not exists alter_routine_load_group $forComputeGroupStr properties ( 'min_cpu_percent'='0');"Review scope: all six changed files and related invocation paths, with two review rounds. No additional production finding was identified. This is source-based review: no build or test was run, and the suggested patch has not been executed. |
|
On F-01: both configurations are reachable in the code, but neither is one this suite runs in, and the suggested patch would break the one it runs in outside cloud mode. I am keeping the fixture as it is.
So I would rate this a Minor about running the suite on a cluster with more than one compute group or resource group, not a Blocker, and leave it out of this PR. |
|
/review |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
/review |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
/review |
There was a problem hiding this comment.
Reviewed PR 70130 at head 72e9189 against the supplied exact-head diff. Static review only: no builds or tests were run.
The production change correctly selects the effective workload group, compute group, and job owner for sequential ALTER validation, and the Kinesis path and replay path match the inspected task behavior. The accepted inline finding is the pre-lock validation race described below. The FE Core locking checkpoint is therefore not fully satisfied for concurrent ALTERs; exception handling and Nereids command normalization had no separate issue, and visible-version/Nereids rule checkpoints are not applicable. The prior general PR comment about the regression fixture was considered from the live discussion and is not duplicated as an inline finding. No user-provided review focus was supplied.
The review did not converge: the semantic and test delegated passes repeatedly failed with model-capacity errors, so their required NO_NEW_VALUABLE_FINDINGS responses could not be obtained. I independently swept all six changed files and related task, manager, validation, replay, and test paths; OA-1 is the only verified new finding. No existing inline blocking comment applies.
Review did not converge after 3 rounds. This automated review cannot pass the review check.
All-subagent convergence could not be established because the semantic and test delegated passes repeatedly failed with model-capacity errors; their NO_NEW_VALUABLE_FINDINGS responses are missing. The main agent independently reviewed every changed file and related path and delivered the verified OA-1 finding at the three-round cap.
…er the job lock ### What problem does this PR solve? Issue Number: None Related PR: apache#70130, apache#69749 Problem Summary: The previous commit checks the workload group an ALTER ROUTINE LOAD leaves the job with before the Kafka and the Kinesis job take their write lock, and RoutineLoadManager does not serialize two ALTERs of one job. Two sessions altering the same paused job at once could therefore each check a different combination and leave one that neither checked: for a job in cg_a with wg_old, which cg_a and cg_b both have, an ALTER moving it to cg_b and one setting wg_a, which only cg_a has, both passed, and the job was left with wg_a in cg_b, which fails every task with the retryable CREATE_TASKS_ERR. The check-before-lock placement came from apache#69749, so on master two concurrent moves, one of which also sets a workload group, already raced this way. It takes cloud mode: outside it no ALTER changes the namespace the workload group is looked up in. Both job types now check first thing under the write lock, before the PAUSED check, so the later of two concurrent ALTERs checks what the earlier one left. The Kafka offset conversion stays before the lock: it sends a metadata request through the group's backends and does not depend on the workload group. So an ALTER with Kafka offsets and a workload group it cannot use now converts the offsets before it fails. The lookup is in memory (WorkloadGroupMgr, the access manager and the compute group names), and the same locked section already writes the edit log; no path takes the job lock while holding one of those. ### Release note None ### Check List (For Author) - Test: Unit Test - RoutineLoadComputeGroupTest#testConcurrentAltersCheckWhatTheEarlierOneLeft: the second ALTER starts from inside the first one's workload group lookup and waits for the job lock; with the check before the lock both were accepted and the job ended up with wg_a in cg_b, now the second fails with "Can not find workload group wg_a in compute croup cg_b" and the job keeps wg_old - 11 routine load test classes, 108 cases, pass - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
run buildall |
|
/review |
There was a problem hiding this comment.
Static review completed at the exact PR head 7571ae7 after two bounded rounds. The new under-lock validation fixes the previously reported concurrent workload-group/compute-group tuple race; that existing P1 thread was independently rechecked and is not reposted. One new P1 remains: workload-group ALTER validation can dereference a null owner on legacy pre-v1.1 routine-load jobs instead of returning a controlled error. The full changed-file set, Kafka and Kinesis mutation/replay paths, effective cloud and non-cloud tuple selection, command analysis, and added tests were reviewed. No additional user focus was supplied. Review evidence is static inspection only; no builds or tests were executed under the review contract. All assigned reviewers returned NO_NEW_VALUABLE_FINDINGS in round 2, so the review converged.
Review converged after 2 round(s).
| if (StringUtils.isEmpty(workloadGroup)) { | ||
| return; | ||
| } | ||
| CreateRoutineLoadInfo.checkWorkloadGroup(getUserIdentity(), workloadGroup, |
There was a problem hiding this comment.
[P1] Guard the legacy null owner before calling checkWorkloadGroup. RoutineLoadJob.userIdentity is documented as null for jobs created before v1.1, and an authorized ALTER can still reach this helper. Passing null into CreateRoutineLoadInfo.checkWorkloadGroup creates a context whose authorization path dereferences the identity (AccessTranslation.subjectOf / Auth.getRolesByUserWithLdap), so a workload-group-only ALTER throws an uncaught NullPointerException instead of a UserException. Reject this case with a controlled error (or apply an established legacy owner fallback) and cover it with a regression test.
There was a problem hiding this comment.
The null owner is real, but only a job created before 1.1 has one, and on master such a job cannot run anyway, so this changes how an ALTER of it fails, not whether the job works. I am keeping the helper as it is.
- The owner has been recorded since 1.1 ([fix](resource-tag) Consider resource tags when assigning tasks for broker & routine load #9492; meta version 110 is "For routine load user info"), and every CREATE records the creating session's identity (
KafkaRoutineLoadJob.java:591,:605; Kinesis jobs only exist since 2026). Master does not load an image below meta version 140 (MetaContext.java:37-41), so such a job reaches master only by staying unfinished through every upgrade since 1.0, and ALTER finds only unfinished jobs. - On master that job already fails every task.
KafkaTaskInfo#createRoutineLoadTaskcallsroutineLoadJob.getUserIdentity().getQualifiedUser()unconditionally (KafkaTaskInfo.java:131, since [fix](load) fix multi table load plan fail after restart master Fe or leader change #53799), and for a single-table jobrePlanfirst passes the same null owner to the sameWorkloadGroupMgr.getWorkloadGroup(KafkaTaskInfo.java:204-215). The scheduler pauses the job with CREATE_TASKS_ERR (RoutineLoadTaskScheduler.java:229-236), andSHOW ROUTINE LOADon its database throws atRoutineLoadJob.java:1763. No ALTER can give the job an owner, so the way out is STOP and CREATE, with or without this PR. - The NPE does not escape the statement:
StmtExecutorreturns it to the client as an error (StmtExecutor.java:1084-1091), the job's write lock is released infinally, nothing is merged and no edit log is written. Master already makes this call with the job's owner for an ALTER that moves a job with a workload group to another compute group ([feature](cloud) Support declaring compute_group on routine load and async MV #69749). - The existing fallbacks give no identity to check USAGE against:
RoutineLoadManager#getAvailableBackendIdspicks backends by replica tag, andComputeGroupBindingUtil#checkComputeGroupBeforeTaskskips its check. Skipping here would accept the workload group unchecked, which is what this PR stops. And a regression test cannot create such a job, since CREATE always records the owner.
So I would rate this a Minor about the error an ALTER returns on a job that cannot run, not a P1, and leave it out of this PR. If jobs without an owner are worth handling, it should be one change that covers SHOW, the tasks and ALTER together.
TPC-H: Total hot run time: 28303 ms |
TPC-DS: Total hot run time: 151419 ms |
ClickBench: Total hot run time: 24.18 s |
924060929
left a comment
There was a problem hiding this comment.
Routine-load tasks resolve workload groups using the job owner and effective compute group. This change validates the resulting configuration under the job write lock, before mutation and edit logging, for both Kafka and Kinesis.
Independent static review found no blocking P0/P1/P2 issue in the FE changes. Tuple selection matches task consumption, concurrent ALTER validation is serialized with mutation, and replay bypasses the new validation. The previous race is fixed. The legacy null-owner case changes error reporting for an already unusable job, and the multi-group fixture limitation does not establish a failure in the inspected pipeline topology; neither warrants blocking this PR.
APPROVE for the reviewed FE scope at 7571ae7. No builds or tests were run. The supplied CI snapshot reports successful P0, cloud_p0 and FE UT checks, but coverage and code-review failures remain; this review does not establish merge readiness.
Automated FE-focused static review (gpt-6.1-sol, high). No build or tests were run.
What problem does this PR solve?
Issue Number: None
Related PR: #69749, #50563, #49960, #53490
Problem Summary:
In short.
ALTER ROUTINE LOAD ... PROPERTIES("workload_group" = "...")accepts any name.CREATE ROUTINE LOADchecks the workload group as the job's owner in the compute group the job runs in, and every task of the job resolves it the same way, so a name that is missing there, or that the owner may not use, is accepted by the ALTER and then fails every task. This PR checks it at ALTER time, as CREATE does.Background. A workload group belongs to a compute group in cloud mode, and to the owner's resource group outside it. Each task looks the job's workload group up as the owner in the job's compute group (
KafkaTaskInfo#rePlan,KinesisTaskInfo#rePlan). Kafka jobs used to check an altered workload group (#50563, in theAlterRoutineLoadStmtpath). The NereidsAlterRoutineLoadCommandpath (#49960) did not carry that check, and #53490 removed the old path. #69749 brought a check back only for an ALTER that moves a Kafka job to another compute group. The comment that delays the check ("we can only get clusterId when alter job") is no longer true: the ALTER has the job at hand. Found while reviewing #69749 (thread 4415694066).The problem, and what it cost. Measured on master
856194ece0ewith unit tests; each ALTER below goes straight to the PAUSED check, so on a paused job it is accepted and written to the edit log:cg_a"workload_group" = "wg_b", which onlycg_bhasAfter RESUME, the first task of each such job fails with
Can not find workload group ..., and the job pauses withCREATE_TASKS_ERR. That code is retryable, so auto resume repeats this with backoff (20 s up to 300 s) for as long as the job exists. The ALTER that caused it reported success.How this PR fixes it.
RoutineLoadJob#checkAlteredWorkloadGroup, called by the Kafka and the Kinesis job first thing under the job's write lock, before the PAUSED check, so that of two ALTERs of one job running at once the later one checks what the earlier one left. The Kafka offset conversion stays before the lock, so an ALTER with Kafka offsets and a workload group it cannot use converts the offsets before it fails. When an ALTER sets a workload group or declares a compute group, it checks the resulting workload group in the resulting compute group, withCreateRoutineLoadInfo.checkWorkloadGroupas CREATE does. The resulting workload group is the one the ALTER sets, else the one the job keeps; the resulting compute group is the one the ALTER declares, else the job's. The check runs as the owner, because the tasks run as the owner. A rejected ALTER changes nothing and writes no edit log. The replay of an ALTER does not go through this path.AlterRoutineLoadCommandthat pointed at the old place, or at the old reason for not checking, are corrected.test_routine_load_alteraltered a job toalter_routine_load_groupwithout creating the group: it worked only because the ALTER did not check, or becausetest_routine_loadhappened to create the group first. It now creates the group, the waytest_routine_loaddoes, and checks that an ALTER to a missing group fails.Results.
Can not find workload group wg_b in compute croup cg_a; the job is unchangedbranch-4.1 and branch-4.2 have the same gap.
Release note
ALTER ROUTINE LOAD now rejects a workload_group that does not exist in the job's compute group (outside cloud mode, the job owner's resource group), or that the job's owner may not use, as CREATE ROUTINE LOAD does. Such an ALTER used to succeed, and then every task of the job failed.
Check List (For Author)
Test
RoutineLoadComputeGroupTestgets 3 cases. In cloud mode, a workload-group-only ALTER is checked in the job's compute group, both for a job running in the group snapshotted at creation and for a job that declares its group. Outside cloud mode, Kafka and Kinesis jobs are checked, and every lookup runs as the owner. A third runs two ALTERs of one paused job at once: the second starts from inside the first one's workload group lookup and waits for the job lock, then fails withCan not find workload group wg_a in compute croup cg_b, and the job keeps wg_old. The workload group mock moves into a shared helper. One case of [feature](cloud) Support declaring compute_group on routine load and async MV #69749 now asserts that an ALTER setting neither property looks nothing up, instead of asserting that a workload-group-only ALTER looks nothing up. The first two fail on master; the third fails with the check before the job lock. 11 routine load test classes, 108 cases, pass. Each part of the fix, broken on its own, fails its case: leaving the Kinesis job unchecked, checking a move in the job's old compute group, checking only an ALTER that declares a compute group, checking as whoever runs the ALTER instead of the owner, ignoring the workload group the ALTER sets, and checking before the job lock. Thetest_routine_load_alterchange needs Kafka, so it was only parsed offline and was not run locally.Behavior changed:
ALTER ROUTINE LOADrejects a workload group that the job could not use, which it used to accept. A workload group now has to be created, and granted to the job's owner, before a job is altered to it, as beforeCREATE ROUTINE LOAD.Does this need documentation?
Check List (For Reviewer who merge this PR)