Repository navigation
[Feature]bind workload group to compute group - #49884
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
f450c7f to
ea78b7d
Compare
| | CREATE ROLE (IF NOT EXISTS)? name=identifierOrText (COMMENT STRING_LITERAL)? #createRole | ||
| | CREATE WORKLOAD GROUP (IF NOT EXISTS)? | ||
| name=identifierOrText properties=propertyClause? #createWorkloadGroup | ||
| name=identifierOrText (FOR computeGroup=identifierOrText)? properties=propertyClause? #createWorkloadGroup |
There was a problem hiding this comment.
至少要增加一个SQL BASED regression test,来保证这个语法正确,包括对应的系统表
| return wg; | ||
| } | ||
|
|
||
| public List<TPipelineWorkloadGroup> getWorkloadGroupByName(UserIdentity userIdentity, String wgFullName) |
There was a problem hiding this comment.
这个接口返回值为啥是一个Thrfit? 另外,我们为啥会是一个list,难道还不唯一?
There was a problem hiding this comment.
之前是唯一的,但是在resource tag的功能下,用户可以指定多个tag,同名的但是不同tag的wg,返回的就是多个
| analyzedJobProperties.put(CreateRoutineLoadStmt.WORKLOAD_GROUP, String.valueOf(wgId)); | ||
| if (!StringUtils.isEmpty(workloadGroup)) { | ||
| long wgId = Env.getCurrentEnv().getWorkloadGroupMgr() | ||
| .getWorkloadGroupByName(ConnectContext.get().getCurrentUserIdentity(), workloadGroup).get(0) |
There was a problem hiding this comment.
这里比较尴尬的点是,最初设计routine load的参数的时候,只能存了一个wg的id,所以目前routineload可能没法实现多tag间的路由,得用别的办法解决。
| Config.enable_advance_next_id = this.enableAdvanceNextId; | ||
| FeConstants.enableInternalSchemaDb = false; | ||
| FeConstants.shouldCreateInternalWorkloadGroup = false; | ||
| FeConstants.dropOldWorkloadGroup = false; |
There was a problem hiding this comment.
这个东西不是控制线程是否运行,这个参数的作用是跑FE ut的时候,跳过这个线程
| .column("MEMORY_LOW_WATERMARK", ScalarType.createVarchar(256)) | ||
| .column("MEMORY_HIGH_WATERMARK", ScalarType.createVarchar(256)) | ||
| .column("TAG", ScalarType.createVarchar(256)) | ||
| .column("COMPUTE_GROUP", ScalarType.createVarchar(256)) |
There was a problem hiding this comment.
这些地方,我们得记得改文档。
另外,这里叫 Compute_GROUP/RESOURCEGROUP 好一些,否则,存算一体下,不好解释
There was a problem hiding this comment.
这么说的话,要改的不止一处啊,所有报错的地方,都得是compute_group/resource group....
| return workloadId.longValue(); | ||
| } | ||
| public List<TPipelineWorkloadGroup> getWorkloadGroup(ConnectContext context) throws UserException { | ||
| String wgName = getWorkloadGroupNameAndCheckPriv(context); |
There was a problem hiding this comment.
我们现在这种check 方式,解决不了这个场景:
CGA 里有wg1,wg2
CGB 里有wg1,wg2
我想让一个用户只有CGA.WG1的权限,有CGB.WG2的权限,但是没有CGB的WG1的权限。
| return oldWgList; | ||
| } | ||
|
|
||
| public void dropOldWorkloadGroup(Set<String> cgSet, WorkloadGroup oldWg) { |
There was a problem hiding this comment.
这个不要叫drop吧,你这个叫convert global wg to per local cg
|
|
||
| // drop old workload group | ||
| nameToWorkloadGroup.remove(oldKey); | ||
| idToWorkloadGroup.remove(oldWg.getId()); |
| tWgList = Env.getCurrentEnv().getWorkloadGroupMgr().getWorkloadGroup(tmpCtx); | ||
| } | ||
| if (tWgList.size() != 0) { | ||
| tExecPlanFragmentParams.setWorkloadGroups(tWgList); |
There was a problem hiding this comment.
fe给be传的一直都是list,只不过之前list集合大小为1,现在可能会出现同名的wg在不同的分组上,比如resource tag就可以指定多个
| throw new AnalysisException( | ||
| WorkloadGroupMgr.DEFAULT_GROUP_NAME | ||
| + " group can not set tag"); | ||
| "tag is deprecated, you can use create workload group [for compute group] as a replacement."); |
There was a problem hiding this comment.
提示词改成 for compute/resource group
| String tagStr = properties.get(WorkloadGroup.TAG); | ||
| if (!StringUtils.isEmpty(tagStr) | ||
| && WorkloadGroupMgr.DEFAULT_GROUP_NAME.equals(workloadGroupName)) { | ||
| if (!StringUtils.isEmpty(tagStr)) { |
There was a problem hiding this comment.
如果tagStr == null,此时78 行的结果是啥?
There was a problem hiding this comment.
字符串为null或者empty都会被StringUtils.isEmpty判定为空
| // todo: add check for backend tag. | ||
| computeGroup = Tag.DEFAULT_BACKEND_TAG.value; | ||
| } else if (Config.isCloudMode()) { | ||
| String originStr = computeGroup; |
There was a problem hiding this comment.
存算一体的时候,不需要check tag的backend 是否存在吗?
| computeGroup = ((CloudSystemInfoService) Env.getCurrentEnv().getClusterInfo()).getCloudClusterIdByName( | ||
| computeGroup); | ||
| if (StringUtils.isEmpty(computeGroup)) { | ||
| throw new UserException("can not find clusterId for " + originStr); |
There was a problem hiding this comment.
这里为啥提示词,变成了clusterId?
另外,所有的报错信息里,一句话的开头,第一个字母大写
| } | ||
|
|
||
| public Set<String> getNames() { | ||
| return Sets.newHashSet(name); |
There was a problem hiding this comment.
不要增加这种函数,他本身就不会有names
在getComputeGroupNames 这个函数里自己包装set 吧
There was a problem hiding this comment.
存算一体是支持指定多个rg的,根据cg.wg找workload group的时候,得有个方法返回多个wg的name,所以ComputeGroup无论如何都得返回一个name的集合
ce3e7e6 to
ca577cb
Compare
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression P0 && UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression P0 && UT Coverage ReportIncrement line coverage Increment coverage report
|
|
run buildall |
TPC-H: Total hot run time: 33677 ms |
TPC-DS: Total hot run time: 192444 ms |
ClickBench: Total hot run time: 29.61 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression P0 && UT Coverage ReportIncrement line coverage Increment coverage report
|
|
run buildall |
TPC-H: Total hot run time: 33733 ms |
TPC-DS: Total hot run time: 193457 ms |
ClickBench: Total hot run time: 29.78 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression P0 && UT Coverage ReportIncrement line coverage Increment coverage report
|
|
run buildall |
TPC-H: Total hot run time: 34158 ms |
TPC-DS: Total hot run time: 185625 ms |
ClickBench: Total hot run time: 29.23 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
|
PR approved by at least one committer and no changes requested. |
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)