Skip to content

[Feature]bind workload group to compute group - #49884

Merged
wangbo merged 4 commits into
apache:masterfrom
wangbo:0402_wg_cg_pr
Apr 28, 2025
Merged

wangbo merged 4 commits into
apache:masterfrom
wangbo:0402_wg_cg_pr

Conversation

@wangbo

@wangbo wangbo commented Apr 8, 2025

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@wangbo
wangbo marked this pull request as draft April 8, 2025 12:24
@wangbo
wangbo force-pushed the 0402_wg_cg_pr branch 2 times, most recently from f450c7f to ea78b7d Compare April 9, 2025 02:35
| CREATE ROLE (IF NOT EXISTS)? name=identifierOrText (COMMENT STRING_LITERAL)? #createRole
| CREATE WORKLOAD GROUP (IF NOT EXISTS)?
name=identifierOrText properties=propertyClause? #createWorkloadGroup
name=identifierOrText (FOR computeGroup=identifierOrText)? properties=propertyClause? #createWorkloadGroup

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

至少要增加一个SQL BASED regression test,来保证这个语法正确,包括对应的系统表

return wg;
}

public List<TPipelineWorkloadGroup> getWorkloadGroupByName(UserIdentity userIdentity, String wgFullName)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个接口返回值为啥是一个Thrfit? 另外,我们为啥会是一个list,难道还不唯一?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

之前是唯一的,但是在resource tag的功能下,用户可以指定多个tag,同名的但是不同tag的wg,返回的就是多个

analyzedJobProperties.put(CreateRoutineLoadStmt.WORKLOAD_GROUP, String.valueOf(wgId));
if (!StringUtils.isEmpty(workloadGroup)) {
long wgId = Env.getCurrentEnv().getWorkloadGroupMgr()
.getWorkloadGroupByName(ConnectContext.get().getCurrentUserIdentity(), workloadGroup).get(0)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里为啥用的是第0个?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里比较尴尬的点是,最初设计routine load的参数的时候,只能存了一个wg的id,所以目前routineload可能没法实现多tag间的路由,得用别的办法解决。

Config.enable_advance_next_id = this.enableAdvanceNextId;
FeConstants.enableInternalSchemaDb = false;
FeConstants.shouldCreateInternalWorkloadGroup = false;
FeConstants.dropOldWorkloadGroup = false;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个改成是一个fe conf

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个东西不是控制线程是否运行,这个参数的作用是跑FE ut的时候,跳过这个线程

.column("MEMORY_LOW_WATERMARK", ScalarType.createVarchar(256))
.column("MEMORY_HIGH_WATERMARK", ScalarType.createVarchar(256))
.column("TAG", ScalarType.createVarchar(256))
.column("COMPUTE_GROUP", ScalarType.createVarchar(256))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这些地方,我们得记得改文档。
另外,这里叫 Compute_GROUP/RESOURCEGROUP 好一些,否则,存算一体下,不好解释

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这么说的话,要改的不止一处啊,所有报错的地方,都得是compute_group/resource group....

return workloadId.longValue();
}
public List<TPipelineWorkloadGroup> getWorkloadGroup(ConnectContext context) throws UserException {
String wgName = getWorkloadGroupNameAndCheckPriv(context);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

我们现在这种check 方式,解决不了这个场景:
CGA 里有wg1,wg2
CGB 里有wg1,wg2

我想让一个用户只有CGA.WG1的权限,有CGB.WG2的权限,但是没有CGB的WG1的权限。

return oldWgList;
}

public void dropOldWorkloadGroup(Set<String> cgSet, WorkloadGroup oldWg) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个不要叫drop吧,你这个叫convert global wg to per local cg


// drop old workload group
nameToWorkloadGroup.remove(oldKey);
idToWorkloadGroup.remove(oldWg.getId());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

如果之前的权限的逻辑不对,这里我们可能还得改权限。

tWgList = Env.getCurrentEnv().getWorkloadGroupMgr().getWorkloadGroup(tmpCtx);
}
if (tWgList.size() != 0) {
tExecPlanFragmentParams.setWorkloadGroups(tWgList);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

为啥传递给BE的是一个wg list?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fe给be传的一直都是list,只不过之前list集合大小为1,现在可能会出现同名的wg在不同的分组上,比如resource tag就可以指定多个

throw new AnalysisException(
WorkloadGroupMgr.DEFAULT_GROUP_NAME
+ " group can not set tag");
"tag is deprecated, you can use create workload group [for compute group] as a replacement.");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

提示词改成 for compute/resource group

String tagStr = properties.get(WorkloadGroup.TAG);
if (!StringUtils.isEmpty(tagStr)
&& WorkloadGroupMgr.DEFAULT_GROUP_NAME.equals(workloadGroupName)) {
if (!StringUtils.isEmpty(tagStr)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

如果tagStr == null,此时78 行的结果是啥?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

字符串为null或者empty都会被StringUtils.isEmpty判定为空

// todo: add check for backend tag.
computeGroup = Tag.DEFAULT_BACKEND_TAG.value;
} else if (Config.isCloudMode()) {
String originStr = computeGroup;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

存算一体的时候,不需要check tag的backend 是否存在吗?

computeGroup = ((CloudSystemInfoService) Env.getCurrentEnv().getClusterInfo()).getCloudClusterIdByName(
computeGroup);
if (StringUtils.isEmpty(computeGroup)) {
throw new UserException("can not find clusterId for " + originStr);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里为啥提示词,变成了clusterId?
另外,所有的报错信息里,一句话的开头,第一个字母大写

}

public Set<String> getNames() {
return Sets.newHashSet(name);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

不要增加这种函数,他本身就不会有names
在getComputeGroupNames 这个函数里自己包装set 吧

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

存算一体是支持指定多个rg的,根据cg.wg找workload group的时候,得有个方法返回多个wg的name,所以ComputeGroup无论如何都得返回一个name的集合

@wangbo
wangbo force-pushed the 0402_wg_cg_pr branch 4 times, most recently from ce3e7e6 to ca577cb Compare April 15, 2025 12:10
@wangbo
wangbo marked this pull request as ready for review April 15, 2025 12:12
@doris-robot

Copy link
Copy Markdown

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 54.39% (14743/27107)
Line Coverage 43.46% (128980/296753)
Region Coverage 42.17% (65882/156220)
Branch Coverage 36.69% (33185/90454)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression P0 && UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 55.31% (14719/26612)
Line Coverage 44.89% (132987/296260)
Region Coverage 41.81% (76359/182616)
Branch Coverage 35.83% (36905/102996)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression P0 && UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.77% (14002/26536)
Line Coverage 42.38% (124485/293769)
Region Coverage 39.57% (71709/181243)
Branch Coverage 33.94% (34665/102148)

@wangbo

wangbo commented Apr 27, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

Copy link
Copy Markdown
TPC-H: Total hot run time: 33677 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5be5abf100e1def87d89c97e83aa45d5bbd9f36b, data reload: false

------ Round 1 ----------------------------------
q1	25915	5239	4979	4979
q2	2053	292	188	188
q3	10381	1248	676	676
q4	10213	990	501	501
q5	7524	2309	2315	2309
q6	179	162	132	132
q7	906	743	622	622
q8	9324	1248	1102	1102
q9	6844	5149	5077	5077
q10	6860	2302	1873	1873
q11	476	284	270	270
q12	345	343	210	210
q13	17769	3692	3050	3050
q14	223	231	218	218
q15	540	492	479	479
q16	436	438	401	401
q17	589	846	368	368
q18	7645	7151	7069	7069
q19	1255	938	544	544
q20	336	337	223	223
q21	4000	2616	2396	2396
q22	1053	1028	990	990
Total cold run time: 114866 ms
Total hot run time: 33677 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5099	5012	5037	5012
q2	232	326	228	228
q3	2169	2639	2256	2256
q4	1352	1771	1404	1404
q5	4344	4366	4351	4351
q6	223	176	124	124
q7	2043	1905	1762	1762
q8	2575	2634	2567	2567
q9	7318	7507	7244	7244
q10	3057	3249	2834	2834
q11	600	518	512	512
q12	699	778	619	619
q13	3646	3975	3338	3338
q14	287	292	266	266
q15	549	517	512	512
q16	476	494	454	454
q17	1124	1541	1417	1417
q18	7795	7660	7476	7476
q19	813	829	974	829
q20	2004	2025	1840	1840
q21	5145	4757	4677	4677
q22	1132	1074	1066	1066
Total cold run time: 52682 ms
Total hot run time: 50788 ms

@doris-robot

Copy link
Copy Markdown
TPC-DS: Total hot run time: 192444 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5be5abf100e1def87d89c97e83aa45d5bbd9f36b, data reload: false

query1	1421	1104	1064	1064
query2	6349	1903	1914	1903
query3	12348	4633	4677	4633
query4	26795	23822	22814	22814
query5	5689	646	450	450
query6	302	192	186	186
query7	4002	510	281	281
query8	297	243	225	225
query9	8546	2547	2545	2545
query10	538	323	264	264
query11	15455	15057	14898	14898
query12	164	107	104	104
query13	1541	528	380	380
query14	9309	6011	6116	6011
query15	206	188	166	166
query16	7562	697	489	489
query17	1098	728	604	604
query18	2043	426	310	310
query19	191	188	164	164
query20	123	125	122	122
query21	209	130	113	113
query22	4558	4534	4297	4297
query23	34340	33608	33304	33304
query24	8419	2420	2434	2420
query25	499	490	382	382
query26	905	277	149	149
query27	2751	514	340	340
query28	4655	2163	2128	2128
query29	680	579	434	434
query30	275	224	181	181
query31	891	847	791	791
query32	77	63	65	63
query33	513	361	305	305
query34	799	913	526	526
query35	812	834	751	751
query36	963	974	886	886
query37	117	101	74	74
query38	4248	4303	4182	4182
query39	1480	1526	1463	1463
query40	213	128	112	112
query41	67	61	57	57
query42	125	112	120	112
query43	514	512	489	489
query44	1358	836	827	827
query45	185	178	172	172
query46	871	1055	670	670
query47	1855	1926	1806	1806
query48	377	415	316	316
query49	740	517	439	439
query50	706	712	417	417
query51	4196	4226	4244	4226
query52	115	114	97	97
query53	236	310	182	182
query54	606	581	523	523
query55	85	79	76	76
query56	284	296	292	292
query57	1163	1202	1155	1155
query58	264	263	270	263
query59	2841	2992	2864	2864
query60	322	320	311	311
query61	128	128	129	128
query62	765	722	672	672
query63	215	182	184	182
query64	4244	1067	710	710
query65	4447	4362	4388	4362
query66	940	416	312	312
query67	15883	15642	15526	15526
query68	8233	912	508	508
query69	519	299	254	254
query70	1171	1078	1080	1078
query71	509	306	295	295
query72	5584	4705	4793	4705
query73	687	632	421	421
query74	8964	9163	8923	8923
query75	4025	3217	2671	2671
query76	3794	1267	771	771
query77	792	373	278	278
query78	9955	10176	9417	9417
query79	2425	803	554	554
query80	592	500	452	452
query81	482	255	221	221
query82	468	126	96	96
query83	244	249	230	230
query84	248	96	80	80
query85	751	356	309	309
query86	347	304	288	288
query87	4461	4442	4275	4275
query88	3893	2225	2217	2217
query89	410	325	285	285
query90	1945	205	210	205
query91	143	140	110	110
query92	80	63	59	59
query93	1959	991	581	581
query94	637	398	306	306
query95	361	290	278	278
query96	508	612	279	279
query97	3146	3283	3114	3114
query98	232	207	201	201
query99	1424	1389	1283	1283
Total cold run time: 283904 ms
Total hot run time: 192444 ms

@doris-robot

Copy link
Copy Markdown
ClickBench: Total hot run time: 29.61 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5be5abf100e1def87d89c97e83aa45d5bbd9f36b, data reload: false

query1	0.03	0.04	0.03
query2	0.12	0.10	0.11
query3	0.25	0.20	0.20
query4	1.59	0.19	0.20
query5	0.61	0.60	0.63
query6	1.17	0.72	0.73
query7	0.02	0.02	0.02
query8	0.05	0.03	0.03
query9	0.58	0.52	0.52
query10	0.57	0.58	0.57
query11	0.16	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.61
query14	1.14	1.19	1.18
query15	0.86	0.84	0.86
query16	0.39	0.39	0.41
query17	1.02	1.04	1.06
query18	0.21	0.20	0.20
query19	1.85	1.79	1.77
query20	0.01	0.01	0.02
query21	15.45	0.93	0.56
query22	0.77	1.36	0.73
query23	14.76	1.41	0.63
query24	7.73	0.75	1.16
query25	0.47	0.20	0.09
query26	0.74	0.17	0.15
query27	0.05	0.04	0.05
query28	9.41	0.88	0.43
query29	12.52	3.91	3.28
query30	0.25	0.09	0.06
query31	2.82	0.60	0.38
query32	3.24	0.55	0.47
query33	3.03	2.99	3.08
query34	15.83	5.18	4.51
query35	4.52	4.53	4.60
query36	0.68	0.50	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.03
query40	0.16	0.13	0.13
query41	0.07	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 104.11 s
Total hot run time: 29.61 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 54.39% (14743/27107)
Line Coverage 43.46% (128971/296745)
Region Coverage 42.17% (65880/156218)
Branch Coverage 36.68% (33177/90450)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression P0 && UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 55.23% (14697/26612)
Line Coverage 44.92% (133082/296252)
Region Coverage 41.83% (76386/182614)
Branch Coverage 35.80% (36867/102992)

@wangbo

wangbo commented Apr 28, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

Copy link
Copy Markdown
TPC-H: Total hot run time: 33733 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 46ec1e55a469834a069ae9f1d99aab44a9b6b90d, data reload: false

------ Round 1 ----------------------------------
q1	25661	5199	5031	5031
q2	2070	274	191	191
q3	10410	1240	700	700
q4	10225	1010	523	523
q5	7507	2390	2281	2281
q6	188	162	134	134
q7	927	753	597	597
q8	9309	1272	1072	1072
q9	6762	5049	5093	5049
q10	6847	2311	1873	1873
q11	471	276	266	266
q12	341	343	204	204
q13	17768	3615	3052	3052
q14	222	214	211	211
q15	553	493	481	481
q16	441	443	396	396
q17	586	858	373	373
q18	7509	7238	7123	7123
q19	1221	945	546	546
q20	340	331	222	222
q21	4015	2651	2447	2447
q22	1074	1022	961	961
Total cold run time: 114447 ms
Total hot run time: 33733 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5188	5139	5083	5083
q2	245	338	239	239
q3	2316	2717	2380	2380
q4	1402	1821	1373	1373
q5	4440	4465	4419	4419
q6	212	168	126	126
q7	1969	1899	1798	1798
q8	2575	2552	2514	2514
q9	7305	7166	7055	7055
q10	3034	3191	2740	2740
q11	582	515	483	483
q12	668	774	610	610
q13	3504	3957	3299	3299
q14	300	309	274	274
q15	527	487	468	468
q16	449	507	478	478
q17	1160	1532	1429	1429
q18	7706	7668	7398	7398
q19	821	807	823	807
q20	1950	1953	1853	1853
q21	5088	4716	4720	4716
q22	1122	1076	1021	1021
Total cold run time: 52563 ms
Total hot run time: 50563 ms

@doris-robot

Copy link
Copy Markdown
TPC-DS: Total hot run time: 193457 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 46ec1e55a469834a069ae9f1d99aab44a9b6b90d, data reload: false

query1	1403	1083	1033	1033
query2	6198	1851	1853	1851
query3	11145	4653	4825	4653
query4	25507	23991	23529	23529
query5	4684	614	435	435
query6	316	221	212	212
query7	3994	501	278	278
query8	291	242	244	242
query9	8487	2580	2566	2566
query10	473	316	255	255
query11	15585	15029	15273	15029
query12	169	110	112	110
query13	1568	518	383	383
query14	9084	6219	6191	6191
query15	204	202	173	173
query16	7596	651	487	487
query17	1159	800	608	608
query18	2081	416	328	328
query19	205	226	181	181
query20	133	122	127	122
query21	209	136	112	112
query22	4359	4444	4360	4360
query23	34691	33642	33661	33642
query24	8805	2456	2426	2426
query25	535	468	411	411
query26	1351	277	158	158
query27	2823	504	327	327
query28	4769	2171	2214	2171
query29	711	580	438	438
query30	267	225	185	185
query31	916	863	779	779
query32	69	66	62	62
query33	540	372	308	308
query34	802	868	522	522
query35	787	828	755	755
query36	979	1012	904	904
query37	116	100	82	82
query38	4342	4341	4093	4093
query39	1485	1431	1474	1431
query40	213	118	105	105
query41	56	55	52	52
query42	118	102	112	102
query43	489	503	497	497
query44	1393	832	842	832
query45	183	177	164	164
query46	840	1046	638	638
query47	1828	1891	1782	1782
query48	381	438	316	316
query49	783	513	415	415
query50	658	691	420	420
query51	4258	4187	4182	4182
query52	108	112	95	95
query53	226	264	190	190
query54	597	616	506	506
query55	83	82	80	80
query56	310	300	297	297
query57	1171	1213	1134	1134
query58	274	266	266	266
query59	2726	2848	2685	2685
query60	352	313	312	312
query61	135	131	146	131
query62	773	722	694	694
query63	238	184	187	184
query64	4122	1070	753	753
query65	4490	4412	4355	4355
query66	951	415	303	303
query67	15712	15813	15299	15299
query68	8680	882	518	518
query69	435	298	264	264
query70	1257	1164	1112	1112
query71	469	328	313	313
query72	5561	4727	4767	4727
query73	677	609	356	356
query74	8846	8857	9023	8857
query75	3662	3223	2721	2721
query76	3404	1212	834	834
query77	751	392	286	286
query78	10073	10097	9304	9304
query79	2268	857	562	562
query80	574	531	448	448
query81	502	251	231	231
query82	451	122	93	93
query83	279	255	234	234
query84	294	99	84	84
query85	854	370	311	311
query86	364	324	301	301
query87	4323	4393	4375	4375
query88	3531	2248	2213	2213
query89	400	318	288	288
query90	1865	213	215	213
query91	144	144	114	114
query92	77	59	58	58
query93	1928	965	582	582
query94	660	394	298	298
query95	374	296	281	281
query96	491	576	275	275
query97	3202	3251	3129	3129
query98	229	217	201	201
query99	1435	1419	1253	1253
Total cold run time: 280366 ms
Total hot run time: 193457 ms

@doris-robot

Copy link
Copy Markdown
ClickBench: Total hot run time: 29.78 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 46ec1e55a469834a069ae9f1d99aab44a9b6b90d, data reload: false

query1	0.03	0.04	0.03
query2	0.12	0.11	0.10
query3	0.26	0.20	0.19
query4	1.59	0.20	0.20
query5	0.60	0.59	0.60
query6	1.17	0.73	0.72
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.57	0.50	0.53
query10	0.58	0.58	0.56
query11	0.16	0.11	0.11
query12	0.15	0.11	0.12
query13	0.61	0.60	0.60
query14	1.20	1.18	1.17
query15	0.88	0.84	0.85
query16	0.38	0.38	0.39
query17	1.03	1.04	1.08
query18	0.21	0.19	0.20
query19	1.84	1.84	1.80
query20	0.02	0.02	0.01
query21	15.41	0.87	0.53
query22	0.75	1.14	0.74
query23	14.89	1.38	0.62
query24	6.71	1.93	0.76
query25	0.52	0.17	0.13
query26	0.60	0.15	0.14
query27	0.05	0.04	0.05
query28	9.96	0.91	0.43
query29	12.60	4.06	3.34
query30	0.25	0.10	0.06
query31	2.82	0.58	0.39
query32	3.22	0.55	0.48
query33	3.08	3.06	3.05
query34	15.84	5.09	4.55
query35	4.53	4.57	4.55
query36	0.67	0.50	0.49
query37	0.09	0.07	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.03
query40	0.16	0.14	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 103.84 s
Total hot run time: 29.78 s

@doris-robot

Copy link
Copy Markdown

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 54.38% (14742/27107)
Line Coverage 43.48% (129022/296740)
Region Coverage 42.18% (65879/156201)
Branch Coverage 36.70% (33188/90432)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression P0 && UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 55.41% (14746/26611)
Line Coverage 44.98% (133258/296256)
Region Coverage 41.91% (76538/182603)
Branch Coverage 35.89% (36965/102982)

@wangbo

wangbo commented Apr 28, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

Copy link
Copy Markdown
TPC-H: Total hot run time: 34158 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3b81a7da501bd6aa27e2b743a6c997b74089da80, data reload: false

------ Round 1 ----------------------------------
q1	26350	5118	5052	5052
q2	2074	277	186	186
q3	10451	1413	727	727
q4	10269	1009	540	540
q5	7982	2395	2370	2370
q6	191	171	137	137
q7	923	747	629	629
q8	9333	1317	1149	1149
q9	6771	5155	5112	5112
q10	6828	2374	1905	1905
q11	447	280	253	253
q12	348	362	223	223
q13	17794	3692	3102	3102
q14	223	227	206	206
q15	537	497	484	484
q16	424	418	369	369
q17	584	871	369	369
q18	7934	7209	7070	7070
q19	1340	968	571	571
q20	339	340	232	232
q21	4293	3443	2485	2485
q22	1072	1040	987	987
Total cold run time: 116507 ms
Total hot run time: 34158 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5141	5085	5101	5085
q2	237	323	222	222
q3	2150	2644	2293	2293
q4	1351	1833	1490	1490
q5	4546	4408	4333	4333
q6	205	183	132	132
q7	1996	1914	1749	1749
q8	2611	2492	2560	2492
q9	7264	7255	7090	7090
q10	2985	3222	2700	2700
q11	576	496	496	496
q12	661	767	613	613
q13	3522	3901	3294	3294
q14	270	307	265	265
q15	534	474	489	474
q16	468	486	446	446
q17	1174	1599	1341	1341
q18	7722	7474	7460	7460
q19	785	812	818	812
q20	2024	2034	1889	1889
q21	5104	4668	4468	4468
q22	1047	1058	1029	1029
Total cold run time: 52373 ms
Total hot run time: 50173 ms

@doris-robot

Copy link
Copy Markdown
TPC-DS: Total hot run time: 185625 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3b81a7da501bd6aa27e2b743a6c997b74089da80, data reload: false

query1	1026	480	487	480
query2	6554	1787	1793	1787
query3	6739	219	224	219
query4	26760	23725	22966	22966
query5	4321	602	435	435
query6	307	208	211	208
query7	4621	485	275	275
query8	280	234	234	234
query9	8604	2566	2567	2566
query10	468	310	293	293
query11	15568	15042	15023	15023
query12	173	119	107	107
query13	1657	526	402	402
query14	8711	6085	6104	6085
query15	199	198	191	191
query16	7241	639	491	491
query17	1205	740	595	595
query18	1995	402	304	304
query19	195	185	169	169
query20	121	119	118	118
query21	213	128	111	111
query22	4092	4260	4029	4029
query23	34102	33075	32995	32995
query24	8450	2424	2438	2424
query25	530	456	387	387
query26	1233	264	148	148
query27	2774	488	335	335
query28	4370	2098	2071	2071
query29	772	538	434	434
query30	270	213	186	186
query31	925	858	750	750
query32	71	66	61	61
query33	551	366	307	307
query34	773	835	533	533
query35	786	789	710	710
query36	947	995	892	892
query37	109	100	76	76
query38	4082	4193	4084	4084
query39	1478	1384	1380	1380
query40	208	116	102	102
query41	56	53	54	53
query42	115	101	104	101
query43	497	491	450	450
query44	1261	801	790	790
query45	179	172	168	168
query46	834	1024	638	638
query47	1756	1826	1741	1741
query48	368	408	296	296
query49	763	517	401	401
query50	646	669	443	443
query51	4155	4159	4019	4019
query52	113	109	95	95
query53	233	260	181	181
query54	580	562	504	504
query55	84	81	83	81
query56	294	314	281	281
query57	1143	1156	1131	1131
query58	262	250	258	250
query59	2604	2701	2582	2582
query60	306	319	306	306
query61	134	132	144	132
query62	795	736	657	657
query63	230	181	182	181
query64	4381	1005	672	672
query65	4369	4212	4254	4212
query66	1137	411	302	302
query67	15802	15631	15414	15414
query68	8015	884	509	509
query69	468	302	271	271
query70	1211	1118	1057	1057
query71	500	310	275	275
query72	5366	4795	4868	4795
query73	731	651	350	350
query74	8984	9169	8943	8943
query75	3780	3192	2733	2733
query76	3681	1178	736	736
query77	782	383	289	289
query78	9961	10119	9292	9292
query79	2783	808	571	571
query80	627	514	433	433
query81	483	266	217	217
query82	480	125	99	99
query83	287	259	225	225
query84	299	103	81	81
query85	777	351	314	314
query86	338	304	293	293
query87	4449	4504	4281	4281
query88	3081	2172	2176	2172
query89	414	316	284	284
query90	1956	204	206	204
query91	146	148	114	114
query92	76	62	60	60
query93	2288	935	576	576
query94	664	413	303	303
query95	380	340	286	286
query96	477	567	272	272
query97	3154	3255	3072	3072
query98	242	210	202	202
query99	1447	1407	1272	1272
Total cold run time: 274905 ms
Total hot run time: 185625 ms

@doris-robot

Copy link
Copy Markdown
ClickBench: Total hot run time: 29.23 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3b81a7da501bd6aa27e2b743a6c997b74089da80, data reload: false

query1	0.03	0.04	0.03
query2	0.13	0.10	0.12
query3	0.26	0.20	0.20
query4	1.59	0.19	0.11
query5	0.56	0.56	0.56
query6	1.19	0.73	0.72
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.57	0.53	0.51
query10	0.56	0.57	0.57
query11	0.16	0.11	0.12
query12	0.14	0.11	0.12
query13	0.61	0.60	0.60
query14	0.78	0.79	0.81
query15	0.88	0.85	0.87
query16	0.37	0.38	0.39
query17	1.04	1.03	1.01
query18	0.22	0.20	0.20
query19	1.88	1.79	1.82
query20	0.01	0.01	0.01
query21	15.40	0.90	0.54
query22	0.76	1.18	0.73
query23	14.90	1.36	0.62
query24	6.93	0.91	1.54
query25	0.52	0.19	0.07
query26	0.62	0.16	0.13
query27	0.05	0.05	0.05
query28	9.06	0.88	0.44
query29	12.55	3.94	3.27
query30	0.25	0.09	0.07
query31	2.83	0.58	0.37
query32	3.23	0.54	0.46
query33	3.01	3.12	3.02
query34	15.85	5.18	4.49
query35	4.59	4.60	4.54
query36	0.66	0.50	0.49
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.03
query42	0.04	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 102.74 s
Total hot run time: 29.23 s

@doris-robot

Copy link
Copy Markdown

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 54.68% (14740/26959)
Line Coverage 43.75% (129081/295050)
Region Coverage 42.48% (65901/155143)
Branch Coverage 37.05% (33209/89642)

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Apr 28, 2025
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wangbo
wangbo merged commit 3c44f18 into apache:master Apr 28, 2025
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants