Repository navigation
[feature](cloud) Support declaring compute_group on routine load and async MV #69749
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
124917f
4ed5856
7611a91
45cda59
4bf82ff
aaff9b5
c9d9730
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -82,6 +82,7 @@ | |
| import org.apache.doris.qe.OriginStatement; | ||
| import org.apache.doris.qe.QeProcessorImpl; | ||
| import org.apache.doris.qe.StmtExecutor; | ||
| import org.apache.doris.resource.computegroup.ComputeGroupBindingUtil; | ||
| import org.apache.doris.rpc.RpcException; | ||
| import org.apache.doris.system.SystemInfoService; | ||
| import org.apache.doris.thrift.TCell; | ||
|
|
@@ -331,6 +332,10 @@ public void run() throws JobException { | |
| } | ||
| mtmvSchemaChangeVersion = mtmv.getSchemaChangeVersion(); | ||
| ConnectContext ctx = MTMVPlanUtil.createMTMVContext(mtmv, MTMVPlanUtil.DISABLE_RULES_WHEN_RUN_MTMV_TASK); | ||
| // The planning done on this context (base table resolution, partition calculation) must see | ||
| // the same compute group the refresh will execute in, otherwise the workload group would be | ||
| // looked up in a different namespace. | ||
| applyComputeGroup(ctx); | ||
| try { | ||
| if (LOG.isDebugEnabled()) { | ||
| String taskSessionContext = ctx.getSessionVariable().toJson().toJSONString(); | ||
|
|
@@ -1663,21 +1668,57 @@ private static void closeExecutionContext(ConnectContext executionContext) { | |
| } | ||
| } | ||
|
|
||
| private void setupComputeGroup(ConnectContext ctx) { | ||
| /** | ||
| * Binds a refresh context to the compute group the refresh runs in, records that group on the task and | ||
| * re-checks the declared one. The declaration is read once, so the group that is checked is the group that | ||
| * is bound: an ALTER MATERIALIZED VIEW landing between two reads could otherwise have the check pass for | ||
| * the new group while this refresh runs in the old one. | ||
| */ | ||
| private void setupComputeGroup(ConnectContext ctx) throws UserException { | ||
| if (!Config.isCloudMode()) { | ||
| computeGroup = FeConstants.null_string; | ||
| return; | ||
| } | ||
| String taskComputeGroup = taskContext.getComputeGroup(); | ||
| if (!Strings.isNullOrEmpty(taskComputeGroup)) { | ||
| ctx.setCloudCluster(taskComputeGroup); | ||
| } | ||
| Optional<String> declaredComputeGroup = applyComputeGroup(ctx); | ||
| try { | ||
| computeGroup = ctx.getCloudCluster(false); | ||
| } catch (ComputeGroupException e) { | ||
| computeGroup = FeConstants.null_string; | ||
| LOG.warn("failed to resolve compute group for mtmv task, taskId: {}", getTaskId(), e); | ||
| } | ||
| checkComputeGroupBeforeTask(ctx, declaredComputeGroup); | ||
| } | ||
|
|
||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [P2] Use one compute-group snapshot for binding and validation
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed in |
||
| // Returns the declaration it read, so that a re-check of the declared group checks the bound one. | ||
| private Optional<String> applyComputeGroup(ConnectContext ctx) { | ||
| // A compute group declared on the MV pins every refresh, automatic or manual, to that group. | ||
| // Only when the MV declares nothing does a manual REFRESH keep borrowing the session's group, | ||
| // which is the behaviour every existing MV keeps. | ||
| Optional<String> declaredComputeGroup = mtmv.getComputeGroup(); | ||
| String effective = declaredComputeGroup.orElse(taskContext.getComputeGroup()); | ||
| if (!Strings.isNullOrEmpty(effective)) { | ||
| // A no-op outside cloud mode. | ||
| ctx.setCloudCluster(effective); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [P2] Keep the declared group authoritative after query hints run An MV with
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is the existing precedence between a statement hint and an object's properties rather than something this PR introduces. |
||
| } | ||
| return declaredComputeGroup; | ||
| } | ||
|
|
||
| /** | ||
| * Re-checks the declared compute group before the refresh runs: it can be dropped and its | ||
| * privileges revoked while the MV exists, and without this the refresh would fail later with an | ||
| * unrelated message. | ||
| * | ||
| * <p>The identity used here is whatever the refresh actually runs as, which today is the | ||
| * hardcoded {@code admin} (see {@link MTMVPlanUtil#createBasicMvContext}). That makes the | ||
| * privilege half of the check always pass; the existence half is what has teeth right now. Once | ||
| * an MV carries a real owner, passing that owner here is the only change needed. | ||
| */ | ||
| private void checkComputeGroupBeforeTask(ConnectContext ctx, Optional<String> declaredComputeGroup) | ||
| throws UserException { | ||
| // Only the explicitly declared compute group is re-checked; an MV that declares none borrows | ||
| // the session's group, which the user never chose for it. | ||
| ComputeGroupBindingUtil.checkComputeGroupBeforeTask(ctx.getCurrentUserIdentity(), | ||
| declaredComputeGroup.orElse(null)); | ||
| } | ||
|
|
||
| private void recordQueryId(String queryId) { | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -131,6 +131,7 @@ public abstract class RoutineLoadJob | |
| protected static final String STAR_STRING = "*"; | ||
|
|
||
| public static final String WORKLOAD_GROUP = "workload_group"; | ||
| public static final String COMPUTE_GROUP = "compute_group"; | ||
|
|
||
| @Getter | ||
| @Setter | ||
|
|
@@ -438,6 +439,13 @@ protected void setOptional(CreateRoutineLoadInfo info) throws UserException { | |
| if (!StringUtils.isEmpty(info.getWorkloadGroupName())) { | ||
| jobProperties.put(WORKLOAD_GROUP, info.getWorkloadGroupName()); | ||
| } | ||
|
|
||
| // Only write the key when the user declared it. An absent key means "not declared" and | ||
| // keeps the existing implicit resolution, which is also what later versions rely on to | ||
| // tell a pinned group apart from an inherited one. | ||
| if (!StringUtils.isEmpty(info.getComputeGroupName())) { | ||
| jobProperties.put(COMPUTE_GROUP, info.getComputeGroupName()); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [P2] Wake the declared group before the first Kafka metadata request With cloud auto-start enabled, a session on live
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed for CREATE and ALTER, and fixed in 4ed5856. After the USAGE and existence checks, the declared group is now started with The scheduler half is not specific to declared groups.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [P2] Preserve the declared group in SHOW CREATE ROUTINE LOAD The new binding is stored here, but
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed, and fixed in The same list also leaves out |
||
| } | ||
| } | ||
|
|
||
| protected void setRoutineLoadDesc(RoutineLoadDesc routineLoadDesc) { | ||
|
|
@@ -550,6 +558,11 @@ public String getWorkloadGroup() { | |
| return jobProperties.get(WORKLOAD_GROUP); | ||
| } | ||
|
|
||
| // The compute group explicitly declared on the job, or null when the user did not declare one. | ||
| public String getDeclaredComputeGroup() { | ||
| return jobProperties.get(COMPUTE_GROUP); | ||
| } | ||
|
|
||
| public JobState getState() { | ||
| return state; | ||
| } | ||
|
|
@@ -797,7 +810,12 @@ public void setComment(String comment) { | |
| } | ||
|
|
||
| public String getCloudCluster() { | ||
| return cloudCluster; | ||
| // An explicitly declared compute group wins over the cluster snapshotted from the session | ||
| // at create time. Keeping this in the getter (instead of writing both places) means every | ||
| // caller follows the declaration, and ALTER only has to update jobProperties, which is | ||
| // already covered by the existing edit log. | ||
| String declared = getDeclaredComputeGroup(); | ||
| return StringUtils.isEmpty(declared) ? cloudCluster : declared; | ||
| } | ||
|
|
||
| public int getSizeOfRoutineLoadTaskInfoList() { | ||
|
|
@@ -1059,13 +1077,16 @@ public TPipelineFragmentParams plan(NereidsStreamLoadPlanner planner, TUniqueId | |
| table.readLock(); | ||
| try { | ||
| if (Config.isCloudMode()) { | ||
| // Use the effective cluster, not the raw snapshot field: this context is what | ||
| // OlapTableSink uses to pick the backends the load writes to. | ||
| String effectiveCluster = getCloudCluster(); | ||
| if (ConnectContext.get() == null) { | ||
| ConnectContext ctx = new ConnectContext(); | ||
| ctx.setThreadLocalInfo(); | ||
| ctx.setCloudCluster(cloudCluster); | ||
| ctx.setCloudCluster(effectiveCluster); | ||
| needCleanCtx = true; | ||
| } else { | ||
| ConnectContext.get().setCloudCluster(cloudCluster); | ||
| ConnectContext.get().setCloudCluster(effectiveCluster); | ||
| } | ||
| ConnectContext.get().setCurrentUserIdentity(this.getUserIdentity()); | ||
| } else { | ||
|
|
@@ -1488,6 +1509,22 @@ public void updateState(JobState jobState, ErrorReason reason, boolean isReplay) | |
| .build()); | ||
| } | ||
|
|
||
| // For failures found by refreshKafkaPartitions(), which update() runs before it takes the job lock. | ||
| // The state is checked again under the lock: a STOP that completed in between must not be turned | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [P2] Stop auto-resume when a paused job loses group USAGE
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed in |
||
| // back into PAUSED, as replay rejects STOPPED -> PAUSED. Only a NEED_SCHEDULE job is paused. A job that | ||
| // is still PAUSED, as during an auto resume attempt, keeps its reason (PAUSED -> PAUSED is rejected too), | ||
| // so a failure whose error code has to be recorded must not be reported from such an attempt. | ||
| protected void pauseIfNeedSchedule(ErrorReason reason) throws UserException { | ||
| writeLock(); | ||
| try { | ||
| if (state == JobState.NEED_SCHEDULE) { | ||
| unprotectUpdateState(JobState.PAUSED, reason, false /* not replay */); | ||
| } | ||
| } finally { | ||
| writeUnlock(); | ||
| } | ||
| } | ||
|
|
||
| protected void unprotectUpdateState(JobState jobState, ErrorReason reason, boolean isReplay) throws UserException { | ||
| checkStateTransform(jobState); | ||
| switch (jobState) { | ||
|
|
@@ -1667,7 +1704,9 @@ public void setCloudCluster(String cloudCluster) { | |
| } | ||
|
|
||
| public String getClusterInfo() { | ||
| return Strings.nullToEmpty(cloudCluster); | ||
| // SHOW ROUTINE LOAD must display the cluster the job actually runs in, i.e. the declared | ||
| // compute group when there is one. | ||
| return Strings.nullToEmpty(getCloudCluster()); | ||
| } | ||
|
|
||
| // check the correctness of commit info | ||
|
|
@@ -1824,6 +1863,9 @@ public String getShowCreateInfo() { | |
| appendProperties(sb, JsonFileFormatProperties.PROP_JSON_ROOT, getJsonRoot(), false); | ||
| appendProperties(sb, LoadCommand.STRICT_MODE, isStrictMode(), false); | ||
| appendProperties(sb, LoadCommand.TIMEZONE, getTimezone(), false); | ||
| // Only a declared group, which follows ALTER. Writing the effective group instead would turn | ||
| // a group inherited from the creating session into a pinned one when the statement is replayed. | ||
| appendProperties(sb, CreateRoutineLoadInfo.COMPUTE_GROUP, getDeclaredComputeGroup(), false); | ||
| appendProperties(sb, LoadCommand.EXEC_MEM_LIMIT, getMemLimit(), true); | ||
| sb.append(")\n"); | ||
| // 6. data_source | ||
|
|
@@ -2023,6 +2065,10 @@ public void gsonPostProcess() throws IOException { | |
| CreateRoutineLoadCommand command = (CreateRoutineLoadCommand) nereidsParser.parseSingle( | ||
| origStmt.originStmt); | ||
| CreateRoutineLoadInfo createRoutineLoadInfo = command.getCreateRoutineLoadInfo(); | ||
| // This re-parse only rebuilds the RoutineLoadDesc; it must not re-check resources | ||
| // that can legitimately have disappeared since the job was created, because the | ||
| // catch below turns any failure into the final CANCELLED state. | ||
| createRoutineLoadInfo.setReplay(true); | ||
| // If tableId is set, resolve the current table name by ID so that | ||
| // table rename / SWAP TABLE won't cause replay to fail with stale name in origStmt. | ||
| if (!isMultiTable && tableId != 0) { | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -30,6 +30,7 @@ | |
| import org.apache.doris.common.util.LogKey; | ||
| import org.apache.doris.common.util.MasterDaemon; | ||
| import org.apache.doris.load.routineload.RoutineLoadJob.JobState; | ||
| import org.apache.doris.resource.computegroup.ComputeGroupBindingUtil; | ||
| import org.apache.doris.system.Backend; | ||
| import org.apache.doris.thrift.BackendService; | ||
| import org.apache.doris.thrift.TNetworkAddress; | ||
|
|
@@ -135,9 +136,23 @@ private void scheduleOneTask(RoutineLoadTaskInfo routineLoadTaskInfo) throws Exc | |
| } | ||
|
|
||
| try { | ||
| if (routineLoadManager.getJob(routineLoadTaskInfo.getJobId()).isFinal()) { | ||
| RoutineLoadJob job = routineLoadManager.getJob(routineLoadTaskInfo.getJobId()); | ||
| if (job.isFinal()) { | ||
| return; | ||
| } | ||
|
|
||
| // The compute group the job declared can be dropped and the owner's USAGE on it revoked | ||
| // while the job keeps running, so it is re-checked before every task. | ||
| // | ||
| // This has to happen here rather than while the task is being built. Backend allocation | ||
| // below resolves the backends through that same compute group, so a missing group would | ||
| // otherwise pause the job with a generic "no available BE found" before this check was | ||
| // ever reached, which is exactly the misleading message it exists to replace. Running it | ||
| // first also keeps a failing check from leaving a transaction behind, because beginTxn | ||
| // has not happened yet. | ||
| ComputeGroupBindingUtil.checkComputeGroupBeforeTask(job.getUserIdentity(), | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [P2] Check the binding before initial Kafka metadata polling For a paused Kafka job without custom kafka_partitions, drop its declared group and RESUME it. RoutineLoadScheduler.process() first calls updateRoutineLoadJob(), which reaches RoutineLoadJob.update() -> refreshKafkaPartitions() -> KafkaUtil before any task is created. The absent group's empty backend list is converted into "no alive backends" and then a PARTITIONS_ERR with Kafka-property/topic troubleshooting text. The job is paused before reaching this check, and auto-resume repeats that misleading diagnostic. Run the declared-group check before the initial metadata lookup as well, while retaining this check for later tasks. The added missing-group unit test directly invokes scheduleOneTask with an existing task, so it bypasses this production path.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed in 4bf82ff. |
||
| job.getDeclaredComputeGroup()); | ||
|
|
||
| // check if topic has more data to consume | ||
| if (!routineLoadTaskInfo.hasMoreDataToConsume()) { | ||
| needScheduleTasksQueue.addLast(routineLoadTaskInfo); | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[P2] Read compute and workload groups as one binding snapshot
createMTMVContext()already reads the MV's workload group, while this later call independently reads its compute group. The refresh worker can readwg_a, then a SQL thread atomically ALTER both properties from(cg_a, wg_a)to(cg_b, wg_b), then the worker readscg_b. Ifwg_aexists only in A, the ensuing workload-group lookup fails in B despite both stored configurations being valid. The task'sMTMVJoblock does not serialize property ALTER, which only takes the MV lock. Both execution context paths have the same separated reads. Snapshot the related properties under one MV read lock and use that snapshot for context setup and checking.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The window is real but very narrow, and its outcome is one failed refresh with an explicit error rather than a wrong result: an ALTER that changes both properties would have to commit between
createMTMVContextreading the workload group andapplyComputeGroupreading the compute group, a few statements apart on the refresh thread. That refresh then fails its workload group lookup in the new group (Can not find workload group ... in compute group ...), and the next one reads a consistent pair. branch-4.x has the same two reads, and this port keeps them: reading both properties under one MV read lock would close the window, but with no wrong result at stake we would rather not diverge from 4.x here.