Skip to content
18 changes: 18 additions & 0 deletions fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
Original file line number Diff line number Diff line change
Expand Up @@ -658,6 +658,24 @@ public Optional<String> getWorkloadGroup() {
}
}

/**
* The compute group explicitly declared on this MV, empty when the user did not declare one.
* An empty result keeps the existing implicit resolution (admin's group for auto refresh,
* the session's group for a manual REFRESH).
*/
public Optional<String> getComputeGroup() {
readMvLock();
try {
if (mvProperties.containsKey(PropertyAnalyzer.PROPERTIES_COMPUTE_GROUP) && !StringUtils
.isEmpty(mvProperties.get(PropertyAnalyzer.PROPERTIES_COMPUTE_GROUP))) {
return Optional.of(mvProperties.get(PropertyAnalyzer.PROPERTIES_COMPUTE_GROUP));
}
return Optional.empty();
} finally {
readMvUnlock();
}
}

public boolean isUseForRewrite() {
readMvLock();
try {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,12 @@ public class CloudRoutineLoadManager extends RoutineLoadManager {
@Override
public void addRoutineLoadJob(RoutineLoadJob routineLoadJob, String dbName, String tableName)
throws UserException {
if (!Strings.isNullOrEmpty(ConnectContext.get().getCloudCluster())) {
// When the job declares a compute group explicitly, that declaration is the binding and the
// session's cluster must not overwrite it.
if (!Strings.isNullOrEmpty(routineLoadJob.getDeclaredComputeGroup())) {
LOG.info("routine load job {} pinned to declared compute group {}",
routineLoadJob.getName(), routineLoadJob.getDeclaredComputeGroup());
} else if (!Strings.isNullOrEmpty(ConnectContext.get().getCloudCluster())) {
routineLoadJob.setCloudCluster(ConnectContext.get().getCloudCluster());
} else {
throw new UserException("cloud cluster is empty, please specify cloud cluster");
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -262,6 +262,7 @@ public class PropertyAnalyzer {
"async_mv.query_rewrite.consistency_relaxed_tables";
public static final String PROPERTIES_REFRESH_PARTITION_NUM = "refresh_partition_num";
public static final String PROPERTIES_WORKLOAD_GROUP = "workload_group";
public static final String PROPERTIES_COMPUTE_GROUP = "compute_group";
public static final String PROPERTIES_PARTITION_SYNC_LIMIT = "partition_sync_limit";
public static final String PROPERTIES_PARTITION_TIME_UNIT = "partition_sync_time_unit";
public static final String PROPERTIES_PARTITION_DATE_FORMAT = "partition_date_format";
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@
import org.apache.doris.qe.OriginStatement;
import org.apache.doris.qe.QeProcessorImpl;
import org.apache.doris.qe.StmtExecutor;
import org.apache.doris.resource.computegroup.ComputeGroupBindingUtil;
import org.apache.doris.rpc.RpcException;
import org.apache.doris.system.SystemInfoService;
import org.apache.doris.thrift.TCell;
Expand Down Expand Up @@ -331,6 +332,10 @@ public void run() throws JobException {
}
mtmvSchemaChangeVersion = mtmv.getSchemaChangeVersion();
ConnectContext ctx = MTMVPlanUtil.createMTMVContext(mtmv, MTMVPlanUtil.DISABLE_RULES_WHEN_RUN_MTMV_TASK);
// The planning done on this context (base table resolution, partition calculation) must see
// the same compute group the refresh will execute in, otherwise the workload group would be
// looked up in a different namespace.
applyComputeGroup(ctx);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Read compute and workload groups as one binding snapshot

createMTMVContext() already reads the MV's workload group, while this later call independently reads its compute group. The refresh worker can read wg_a, then a SQL thread atomically ALTER both properties from (cg_a, wg_a) to (cg_b, wg_b), then the worker reads cg_b. If wg_a exists only in A, the ensuing workload-group lookup fails in B despite both stored configurations being valid. The task's MTMVJob lock does not serialize property ALTER, which only takes the MV lock. Both execution context paths have the same separated reads. Snapshot the related properties under one MV read lock and use that snapshot for context setup and checking.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The window is real but very narrow, and its outcome is one failed refresh with an explicit error rather than a wrong result: an ALTER that changes both properties would have to commit between createMTMVContext reading the workload group and applyComputeGroup reading the compute group, a few statements apart on the refresh thread. That refresh then fails its workload group lookup in the new group (Can not find workload group ... in compute group ...), and the next one reads a consistent pair. branch-4.x has the same two reads, and this port keeps them: reading both properties under one MV read lock would close the window, but with no wrong result at stake we would rather not diverge from 4.x here.

try {
if (LOG.isDebugEnabled()) {
String taskSessionContext = ctx.getSessionVariable().toJson().toJSONString();
Expand Down Expand Up @@ -1663,21 +1668,57 @@ private static void closeExecutionContext(ConnectContext executionContext) {
}
}

private void setupComputeGroup(ConnectContext ctx) {
/**
* Binds a refresh context to the compute group the refresh runs in, records that group on the task and
* re-checks the declared one. The declaration is read once, so the group that is checked is the group that
* is bound: an ALTER MATERIALIZED VIEW landing between two reads could otherwise have the check pass for
* the new group while this refresh runs in the old one.
*/
private void setupComputeGroup(ConnectContext ctx) throws UserException {
if (!Config.isCloudMode()) {
computeGroup = FeConstants.null_string;
return;
}
String taskComputeGroup = taskContext.getComputeGroup();
if (!Strings.isNullOrEmpty(taskComputeGroup)) {
ctx.setCloudCluster(taskComputeGroup);
}
Optional<String> declaredComputeGroup = applyComputeGroup(ctx);
try {
computeGroup = ctx.getCloudCluster(false);
} catch (ComputeGroupException e) {
computeGroup = FeConstants.null_string;
LOG.warn("failed to resolve compute group for mtmv task, taskId: {}", getTaskId(), e);
}
checkComputeGroupBeforeTask(ctx, declaredComputeGroup);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Use one compute-group snapshot for binding and validation

setupComputeGroup() calls applyComputeGroup(), which reads mtmv.getComputeGroup() and installs that value in the context, then checkComputeGroupBeforeTask() reads the MV property again. ALTER MATERIALIZED VIEW can update the property between those two read locks, so a task can validate cg_b while its context remains bound to cg_a and the refresh executes there. Capture the declared/effective group once and use that snapshot for both context setup and the pre-task check (or validate the context's actual group).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c9d9730047e. The race is real but narrow, and its effect is smaller than described. The refresh ran in the group that was declared when its context was bound, which is a valid order for an ALTER that lands meanwhile. The only problem was that the re-check could look at the new group instead. That shows up when the old group is dropped at the same time: the refresh then failed later, at backend selection, with a generic error instead of Compute group '<name>' not found.. No data was affected. setupComputeGroup() now reads the declaration once, binds and records it, and re-checks that same value. Both refresh paths go through it, and the REFRESH ... WITH DRY RUN context already worked this way. The new MTMVComputeGroupTest#testRefreshContextChecksTheGroupItBinds makes the first read return a dropped group and any later read an existing one. It expects a single read and a check that fails for the bound group, and it fails when the check reads the property again.

// Returns the declaration it read, so that a re-check of the declared group checks the bound one.
private Optional<String> applyComputeGroup(ConnectContext ctx) {
// A compute group declared on the MV pins every refresh, automatic or manual, to that group.
// Only when the MV declares nothing does a manual REFRESH keep borrowing the session's group,
// which is the behaviour every existing MV keeps.
Optional<String> declaredComputeGroup = mtmv.getComputeGroup();
String effective = declaredComputeGroup.orElse(taskContext.getComputeGroup());
if (!Strings.isNullOrEmpty(effective)) {
// A no-op outside cloud mode.
ctx.setCloudCluster(effective);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep the declared group authoritative after query hints run

An MV with PROPERTIES('compute_group'='cg_a') AS SELECT /*+ SET_VAR(compute_group='cg_b') */ k1 FROM base still refreshes in B. For the partition path, the plan is InsertOverwrite -> LogicalSelectHint(SET_VAR(cg_b)) -> Project -> Scan: setup records/checks A, then EliminateLogicalSelectHint reapplies the hint during command analysis and overwrites the session field consumed by CloudCoordinator.prepare(). In the IVM path, parsing the query after setup also overwrites it. The task reports A although execution uses B. Reject conflicting hints or preserve the binding independently of hint-mutated session state, and validate/report the group the coordinator actually consumes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the existing precedence between a statement hint and an object's properties rather than something this PR introduces. compute_group is an alias of the cloud_cluster session variable, and a SET_VAR hint in the MV's own query is applied during analysis (EliminateLogicalSelectHint -> setVarOnceInSql), after the task has prepared its context. On master the same hint already overrides the group of an MV that declares none, and workload_group behaves the same way: MTMVPlanUtil.createMTMVContext copies the MV property into the session, and a SET_VAR(workload_group=...) in the query overrides it while SHOW keeps reporting the property; the compute group the task records is the declared one in the same way. The hint is part of the MV definition its owner wrote, so the refresh following it is consistent with every other session variable. Making the declaration win, or rejecting conflicting hints at CREATE, would be a new rule for hints in MV SQL that should cover workload_group too, which is outside this port. branch-4.x behaves the same.

}
return declaredComputeGroup;
}

/**
* Re-checks the declared compute group before the refresh runs: it can be dropped and its
* privileges revoked while the MV exists, and without this the refresh would fail later with an
* unrelated message.
*
* <p>The identity used here is whatever the refresh actually runs as, which today is the
* hardcoded {@code admin} (see {@link MTMVPlanUtil#createBasicMvContext}). That makes the
* privilege half of the check always pass; the existence half is what has teeth right now. Once
* an MV carries a real owner, passing that owner here is the only change needed.
*/
private void checkComputeGroupBeforeTask(ConnectContext ctx, Optional<String> declaredComputeGroup)
throws UserException {
// Only the explicitly declared compute group is re-checked; an MV that declares none borrows
// the session's group, which the user never chose for it.
ComputeGroupBindingUtil.checkComputeGroupBeforeTask(ctx.getCurrentUserIdentity(),
declaredComputeGroup.orElse(null));
}

private void recordQueryId(String queryId) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,7 @@ public abstract class RoutineLoadJob
protected static final String STAR_STRING = "*";

public static final String WORKLOAD_GROUP = "workload_group";
public static final String COMPUTE_GROUP = "compute_group";

@Getter
@Setter
Expand Down Expand Up @@ -438,6 +439,13 @@ protected void setOptional(CreateRoutineLoadInfo info) throws UserException {
if (!StringUtils.isEmpty(info.getWorkloadGroupName())) {
jobProperties.put(WORKLOAD_GROUP, info.getWorkloadGroupName());
}

// Only write the key when the user declared it. An absent key means "not declared" and
// keeps the existing implicit resolution, which is also what later versions rely on to
// tell a pinned group apart from an inherited one.
if (!StringUtils.isEmpty(info.getComputeGroupName())) {
jobProperties.put(COMPUTE_GROUP, info.getComputeGroupName());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Wake the declared group before the first Kafka metadata request

With cloud auto-start enabled, a session on live cg_a can declare an existing but SUSPENDED cg_b. Validation accepts its name, and this property immediately makes Kafka metadata requests use B, but none of that path calls waitForAutoStart(B) as USE @B does. A CREATE with custom partitions therefore fails during its initial Kafka metadata/offset lookup with no live backends; otherwise the scheduler's partition lookup pauses the job and repeatedly retries without ever requesting B's resume. Activate the declared target before the metadata/backend lookup, without changing the caller's session group or waiting under the job lock.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed for CREATE and ALTER, and fixed in 4ed5856. After the USAGE and existence checks, the declared group is now started with CloudSystemInfoService.waitForAutoStart (ComputeGroupBindingUtil.activateDeclaredComputeGroup), the third step USE @<group> takes in CloudEnv.changeCloudCluster. It runs on the statement thread before the job lock or any Kafka metadata request, does not change the caller's session group, and is not called on the metadata load path. It is not part of validateDeclaredComputeGroup, which async MVs share: an MV sends nothing to its group at CREATE, and its refresh resolves backends through getBackendsByCurrentCluster, which already starts the group it binds.

The scheduler half is not specific to declared groups. CloudRoutineLoadManager.getAvailableBackendIds and KafkaUtil.getBackendIdsForMetaRequest resolve backends by name and never start a suspended group on master, so a job whose implicit group is suspended after creation already pauses and retries without resuming it. Changing that affects every cloud routine load job and is out of scope for this port. testCreateWakesDeclaredComputeGroup and testAlterWakesDeclaredComputeGroup cover the two call sites, and the metadata load test checks that replay starts nothing.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve the declared group in SHOW CREATE ROUTINE LOAD

The new binding is stored here, but getShowCreateInfo() emits a fixed property list that never includes compute_group. SHOW CREATE ROUTINE LOAD calls that method, so creating or ALTERing a job to cg_b, exporting its DDL, and recreating it from a session on cg_a silently creates an unpinned job running in A. Emit the current getDeclaredComputeGroup() in that SQL when nonempty, preserving the absent-key behavior for jobs without a declaration, and cover the round trip after ALTER.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed, and fixed in 7611a91f1d5. The gap is in getShowCreateInfo(), which this PR did not change. Its property list is the same on master and branch-4.x and has no entry for compute_group, while SHOW CREATE MATERIALIZED VIEW renders every MV property, so the MV half already round-trips. On master no job could declare a group and the statement never carried the implicit one, so a job recreated from it has always run in the recreating session's group. For a job that declares its group, though, losing the declaration on export defeats the point of declaring it. The statement now includes compute_group from getDeclaredComputeGroup(), which follows ALTER, and only when a group is declared. Jobs without a declaration keep their current output (RoutineLoadJobTest#testGetShowCreateInfo is unchanged); writing the effective group instead would turn an inherited group into a pinned one when the statement is replayed. RoutineLoadJobTest#testGetShowCreateInfoCarriesDeclaredComputeGroup checks the statement of a job that declares cg_a and is then altered to cg_b, and test_routine_load_compute_group now also checks SHOW CREATE ROUTINE LOAD after its ALTER.

The same list also leaves out workload_group and seven other properties CREATE accepts (send_batch_parallelism, load_to_single_tablet, partial_update_new_key_behavior, unique_key_update_mode, enclose, escape, empty_field_as_null). That predates this PR and affects every routine load job, so it is left to a separate change.

}
}

protected void setRoutineLoadDesc(RoutineLoadDesc routineLoadDesc) {
Expand Down Expand Up @@ -550,6 +558,11 @@ public String getWorkloadGroup() {
return jobProperties.get(WORKLOAD_GROUP);
}

// The compute group explicitly declared on the job, or null when the user did not declare one.
public String getDeclaredComputeGroup() {
return jobProperties.get(COMPUTE_GROUP);
}

public JobState getState() {
return state;
}
Expand Down Expand Up @@ -797,7 +810,12 @@ public void setComment(String comment) {
}

public String getCloudCluster() {
return cloudCluster;
// An explicitly declared compute group wins over the cluster snapshotted from the session
// at create time. Keeping this in the getter (instead of writing both places) means every
// caller follows the declaration, and ALTER only has to update jobProperties, which is
// already covered by the existing edit log.
String declared = getDeclaredComputeGroup();
return StringUtils.isEmpty(declared) ? cloudCluster : declared;
}

public int getSizeOfRoutineLoadTaskInfoList() {
Expand Down Expand Up @@ -1059,13 +1077,16 @@ public TPipelineFragmentParams plan(NereidsStreamLoadPlanner planner, TUniqueId
table.readLock();
try {
if (Config.isCloudMode()) {
// Use the effective cluster, not the raw snapshot field: this context is what
// OlapTableSink uses to pick the backends the load writes to.
String effectiveCluster = getCloudCluster();
if (ConnectContext.get() == null) {
ConnectContext ctx = new ConnectContext();
ctx.setThreadLocalInfo();
ctx.setCloudCluster(cloudCluster);
ctx.setCloudCluster(effectiveCluster);
needCleanCtx = true;
} else {
ConnectContext.get().setCloudCluster(cloudCluster);
ConnectContext.get().setCloudCluster(effectiveCluster);
}
ConnectContext.get().setCurrentUserIdentity(this.getUserIdentity());
} else {
Expand Down Expand Up @@ -1488,6 +1509,22 @@ public void updateState(JobState jobState, ErrorReason reason, boolean isReplay)
.build());
}

// For failures found by refreshKafkaPartitions(), which update() runs before it takes the job lock.
// The state is checked again under the lock: a STOP that completed in between must not be turned

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Stop auto-resume when a paused job loses group USAGE

pauseIfNeedSchedule() only records the supplied reason when state == NEED_SCHEDULE. An already PAUSED Kafka job with a retryable reason and no custom partitions enters updateKafkaPartitions() through needAutoResume(), but a revoked owner USAGE throws CANNOT_RESUME_ERR while this helper does nothing because the state is still PAUSED. The old retryable pauseReason remains, so ScheduleRule keeps auto-resuming and repeating the failed check. Preserve the lock/STOP guard while recording the non-retryable reason (or otherwise disabling auto-resume) for this path.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c9d9730047e. Confirmed: auto resume runs the partition refresh while the job is still PAUSED, and pauseIfNeedSchedule() only pauses a NEED_SCHEDULE job (PAUSED -> PAUSED is rejected, on replay too), so a failing check there kept the old, retryable reason. This came with the refresh check in 4bf82ff4a9d. Before that commit, and on branch-4.x, which has no check in the refresh, the attempt reached the task scheduler's check and recorded CANNOT_RESUME_ERR. The refresh now skips the compute group check for an auto resume attempt. Once the metadata request succeeds, the attempt moves the job to NEED_SCHEDULE, and the check RoutineLoadTaskScheduler runs before the first task pauses the job with CANNOT_RESUME_ERR under the job lock, which stops auto resume. A manual RESUME leaves the job in NEED_SCHEDULE, so the refresh still runs the check then, and a dropped group is still reported as Compute group '<name>' not found.. The comments at the check and at pauseIfNeedSchedule() now state this contract. The new RoutineLoadComputeGroupTest#testAutoResumeLeavesRevokedUsageToTheTaskCheck drives refreshKafkaPartitions(true) for a paused job whose owner lost USAGE, then schedules its first task. It checks that the job ends up paused with CANNOT_RESUME_ERR and is not auto resumed, and it fails when the refresh runs the check for the attempt.

// back into PAUSED, as replay rejects STOPPED -> PAUSED. Only a NEED_SCHEDULE job is paused. A job that
// is still PAUSED, as during an auto resume attempt, keeps its reason (PAUSED -> PAUSED is rejected too),
// so a failure whose error code has to be recorded must not be reported from such an attempt.
protected void pauseIfNeedSchedule(ErrorReason reason) throws UserException {
writeLock();
try {
if (state == JobState.NEED_SCHEDULE) {
unprotectUpdateState(JobState.PAUSED, reason, false /* not replay */);
}
} finally {
writeUnlock();
}
}

protected void unprotectUpdateState(JobState jobState, ErrorReason reason, boolean isReplay) throws UserException {
checkStateTransform(jobState);
switch (jobState) {
Expand Down Expand Up @@ -1667,7 +1704,9 @@ public void setCloudCluster(String cloudCluster) {
}

public String getClusterInfo() {
return Strings.nullToEmpty(cloudCluster);
// SHOW ROUTINE LOAD must display the cluster the job actually runs in, i.e. the declared
// compute group when there is one.
return Strings.nullToEmpty(getCloudCluster());
}

// check the correctness of commit info
Expand Down Expand Up @@ -1824,6 +1863,9 @@ public String getShowCreateInfo() {
appendProperties(sb, JsonFileFormatProperties.PROP_JSON_ROOT, getJsonRoot(), false);
appendProperties(sb, LoadCommand.STRICT_MODE, isStrictMode(), false);
appendProperties(sb, LoadCommand.TIMEZONE, getTimezone(), false);
// Only a declared group, which follows ALTER. Writing the effective group instead would turn
// a group inherited from the creating session into a pinned one when the statement is replayed.
appendProperties(sb, CreateRoutineLoadInfo.COMPUTE_GROUP, getDeclaredComputeGroup(), false);
appendProperties(sb, LoadCommand.EXEC_MEM_LIMIT, getMemLimit(), true);
sb.append(")\n");
// 6. data_source
Expand Down Expand Up @@ -2023,6 +2065,10 @@ public void gsonPostProcess() throws IOException {
CreateRoutineLoadCommand command = (CreateRoutineLoadCommand) nereidsParser.parseSingle(
origStmt.originStmt);
CreateRoutineLoadInfo createRoutineLoadInfo = command.getCreateRoutineLoadInfo();
// This re-parse only rebuilds the RoutineLoadDesc; it must not re-check resources
// that can legitimately have disappeared since the job was created, because the
// catch below turns any failure into the final CANCELLED state.
createRoutineLoadInfo.setReplay(true);
// If tableId is set, resolve the current table name by ID so that
// table rename / SWAP TABLE won't cause replay to fail with stale name in origStmt.
if (!isMultiTable && tableId != 0) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -533,8 +533,9 @@ public long getAvailableBeForTask(long jobId, long previousBeId) throws UserExce
if (availableBeIds.isEmpty()) {
RoutineLoadJob job = getJob(jobId);
if (job != null) {
String msg = "no available BE found for job " + jobId + ", cluster Name {}, " + job.getCloudCluster()
+ "please check the BE status and user's cluster or tags";
String msg = "no available BE found for job " + jobId + ", cluster name: "
+ job.getCloudCluster()
+ ", please check the BE status and user's cluster or tags";
job.updateState(RoutineLoadJob.JobState.PAUSED,
new ErrorReason(InternalErrorCode.INTERNAL_ERR, msg), false /* not replay */);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@
import org.apache.doris.common.util.LogKey;
import org.apache.doris.common.util.MasterDaemon;
import org.apache.doris.load.routineload.RoutineLoadJob.JobState;
import org.apache.doris.resource.computegroup.ComputeGroupBindingUtil;
import org.apache.doris.system.Backend;
import org.apache.doris.thrift.BackendService;
import org.apache.doris.thrift.TNetworkAddress;
Expand Down Expand Up @@ -135,9 +136,23 @@ private void scheduleOneTask(RoutineLoadTaskInfo routineLoadTaskInfo) throws Exc
}

try {
if (routineLoadManager.getJob(routineLoadTaskInfo.getJobId()).isFinal()) {
RoutineLoadJob job = routineLoadManager.getJob(routineLoadTaskInfo.getJobId());
if (job.isFinal()) {
return;
}

// The compute group the job declared can be dropped and the owner's USAGE on it revoked
// while the job keeps running, so it is re-checked before every task.
//
// This has to happen here rather than while the task is being built. Backend allocation
// below resolves the backends through that same compute group, so a missing group would
// otherwise pause the job with a generic "no available BE found" before this check was
// ever reached, which is exactly the misleading message it exists to replace. Running it
// first also keeps a failing check from leaving a transaction behind, because beginTxn
// has not happened yet.
ComputeGroupBindingUtil.checkComputeGroupBeforeTask(job.getUserIdentity(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Check the binding before initial Kafka metadata polling

For a paused Kafka job without custom kafka_partitions, drop its declared group and RESUME it. RoutineLoadScheduler.process() first calls updateRoutineLoadJob(), which reaches RoutineLoadJob.update() -> refreshKafkaPartitions() -> KafkaUtil before any task is created. The absent group's empty backend list is converted into "no alive backends" and then a PARTITIONS_ERR with Kafka-property/topic troubleshooting text. The job is paused before reaching this check, and auto-resume repeats that misleading diagnostic. Run the declared-group check before the initial metadata lookup as well, while retaining this check for later tasks. The added missing-group unit test directly invokes scheduleOneTask with an existing task, so it bypasses this production path.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 4bf82ff. KafkaRoutineLoadJob.updateKafkaPartitions() now runs ComputeGroupBindingUtil.checkComputeGroupBeforeTask before the metadata request. After RESUME, a job whose declared group is gone is paused with Compute group '<name>' not found., the pause the task scheduler gives, instead of PARTITIONS_ERR with the Kafka advice. That pause keeps INTERNAL_ERR, so auto resume picks the job up once the group is back, and a revoked USAGE keeps CANNOT_RESUME_ERR. In other states the refresh only logs, as it does for a failed metadata request. It returns instead of throwing, because RoutineLoadManager.updateRoutineLoadJob() updates every job in one loop. The new RoutineLoadComputeGroupTest#testDroppedComputeGroupPausesJobBeforeKafkaMetadataRequest drives refreshKafkaPartitions for a NEED_SCHEDULE job with a dropped group and checks that no metadata request is sent and that the pause reason names the group. It fails without the check.

job.getDeclaredComputeGroup());

// check if topic has more data to consume
if (!routineLoadTaskInfo.hasMoreDataToConsume()) {
needScheduleTasksQueue.addLast(routineLoadTaskInfo);
Expand Down
Loading
Loading