Skip to content

[Fix](INFO) Explain workload group query limits in memory erros - #68589

Open
linrrzqqq wants to merge 1 commit into
apache:masterfrom
linrrzqqq:wg-err-msg
Open

linrrzqqq wants to merge 1 commit into
apache:masterfrom
linrrzqqq:wg-err-msg

Conversation

@linrrzqqq

@linrrzqqq linrrzqqq commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

When a workload group applies a lower per-query memory limit, the allocation error still suggests raising exec_mem_limit. Record the workload group query limit alongside the effective tracker limit, then use that information to give the relevant advice in the error message. The existing check_limit() decision logic is unchanged.

Release note

Query memory limit errors now identify when the workload group query limit is the active constraint and suggest adjusting the workload group limit or reducing query memory use.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

int64_t workload_group_cap = slot_memory_policy == TWgSlotMemoryPolicy::NONE
? wg_mem_limit
: query_weighted_mem_limit;
resource_ctx->memory_context()->set_mem_limit_from_workload_group(effective_limit,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

你这么改是不行的。
我们现在调用tracker_limit_exceeded_str 这个方法的时候,实际只检查了query 的memlimit,并没有检查workload group的memlimit。
我们workload group 的memlimit 实际是异步生效的,通过后台线程定期刷新来更新query 的memlimit。
你现在的改法把workload group的limit 传递进去,那么后续比如还想打印process 的,就会传递茫茫多的参数。

这块可能的问题是,我们的memtracker 不是一个树状结构,比如 query memtracker----parent ---> workload group memtracker --parent--> process memtracker。

linrrzqqq added a commit to linrrzqqq/doris that referenced this pull request Oct 8, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#68589

Problem Summary: Task, workload group and process limits use separate check
and reservation paths, making their relationships and limiting budgets
hard to trace. Add a MemoryLimit hierarchy with selected-node checks,
recursive reservations and rollback, and ancestor diagnostics. Attach
query and write trackers to their workload group while temporary tracker
switches use the thread's workload group without reparenting shared trackers.
Preserve existing memory accounting, check policies and the dedicated
process/task fast paths for ordinary allocations.

### Release note

Query memory-limit diagnostics include workload group and process ancestors.

### Check List (For Author)

- Test: ASAN BE build and 83 unit tests across MemTrackerTest,
  ThreadMemTrackerMgrTest, AllocatorTest, WorkloadGroupManagerTest,
  WorkloadSchedPolicyTest and PipelineTaskTest passed. Header hygiene and
  clang-format v16 passed. Optimized LLVM IR confirms ordinary allocation
  checks avoid tree dispatch. clang-tidy retains pre-existing diagnostics
  outside the modified lines.
- Behavior changed: Yes, memory-limit diagnostics show ancestor budgets;
  allocation/reservation thresholds and check policies are preserved.
- Does this need documentation: No.
@linrrzqqq

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Complete static review of PR #68589 at be58ba4 (all 16 changed files). Request changes for one P1: the new MemoryLimit parent field bypasses Doris's libc++ compatible atomic shared pointer and breaks the macOS BE build. The existing inline thread 4128931042 about workload-group/query-limit diagnostic design was considered and is not reposted; it is not a separately confirmed P0/P1 blocker. No additional user review focus was supplied.

Critical checkpoints:

  • Goal and proof: The changed error path now prints the task, workload-group, and process limit tree for a query/load tracker with a group parent. Added BE unit tests inspect the diagnostic, local checks, reservation masks, rollback, sibling queries, and binding order. They were read only; no test or end-to-end SQL run proves behavior here.
  • Scope and clarity: The new hierarchy and reservation refactor are broader than the diagnostic change described by the PR. The new type is gathered by the runtime CMake glob, but its raw atomic shared pointer is a concrete platform compatibility failure (inline P1).
  • Concurrency and lifecycle: Worker-thread reservations still account task, attached workload group, then process, with rollback on failure. The parent is bound during query/load context setup; the workload-group registry holds weak resource contexts, and the diagnostic group string takes a shared lock. No new counter imbalance, ownership cycle, or deadlock was substantiated. The group's preexisting admission check remains non-atomic across load and add.
  • Configuration, compatibility, and parallel paths: No new config, wire/storage format, transaction, data-write, or FE-to-BE variable is introduced. Existing workload-group limit changes remain asynchronous. Query, load, write-sibling, tracker-only attachment, and temporary-limiter paths were inspected. The libc++ target is broken as noted above; release ELF/glibc compatibility could not be audited because this review forbids builds.
  • Conditions, performance, and observability: Task-only allocation checks remain inline; recursive ancestor checks and tree formatting occur in explicit checks, reservations, or error paths. The check-mask and failure rollback paths were traced, with no additional substantiated issue. The error includes the current effective task limit and group/process context; existing discussion about more specific advice remains on the PR.
  • Test and style evidence: The new BE tests cover important boundaries but are unexecuted here, and no end-to-end negative case was added. git diff --check passed; the pure-text BE build-hygiene gate passed. This is a static-only review, with no build or runtime validation.

Review completion: two rounds of normal and risk-focused review converged on this single new inline finding, followed by a full changed-file and unresolved-candidate sweep. All other suspicious points were dismissed with code evidence or deduplicated against the existing thread.

Comment thread be/src/runtime/memory/memory_limit.h Resolved
### What problem does this PR solve?

Issue Number: None

Related PR: apache#68589

Problem Summary: Task, workload group and process limits use separate check
and reservation paths, making their relationships and limiting budgets
hard to trace. Add a MemoryLimit hierarchy with selected-node checks,
recursive reservations and rollback, and ancestor diagnostics. Attach
query and write trackers to their workload group while temporary tracker
switches use the thread's workload group without reparenting shared trackers.
Preserve existing memory accounting, check policies and the dedicated
process/task fast paths for ordinary allocations.

Use the existing atomic_shared_ptr wrapper for parent storage so libc++
builds compile while preserving atomic load/store semantics.

### Release note

Query memory-limit diagnostics include workload group and process ancestors.

### Check List (For Author)

- Test: ASAN BE build and 85 unit tests across MemTrackerTest,
  ThreadMemTrackerMgrTest, AllocatorTest, WorkloadGroupManagerTest,
  WorkloadSchedPolicyTest and PipelineTaskTest passed. Header hygiene and
  clang-format v16 passed. The MemoryLimit header and implementation
  passed libc++ compilation checks; the modified header passed clang-tidy. Optimized LLVM IR confirms ordinary allocation
  checks avoid tree dispatch. clang-tidy retains pre-existing diagnostics
  outside the modified lines.
- Behavior changed: Yes, memory-limit diagnostics show ancestor budgets;
  allocation/reservation thresholds and check policies are preserved.
- Does this need documentation: No.
@linrrzqqq

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Complete static review of apache/doris PR #68589 at head ff3c1dd. I found no new substantiated inline issue. All 16 changed files were covered by the main review, two normal full-review passes, and a separate risk-focused pass; all three passes returned NO_NEW_VALUABLE_FINDINGS in round 1. No candidate remains unresolved. There was no additional user-provided review focus.

Critical checkpoints:

  • Goal, scope, and tests: The change attaches the query task tracker to a workload-group/process limit hierarchy so a query allocation error can show its effective task limit and group context. The added query_error_includes_memory_limit_ancestors BE test asserts the group appears in the error and the old exec_mem_limit-only advice is absent. The hierarchy is broader than the PR description, but the changes are cohesive. These are inspected tests, not executed validation.
  • Concurrency and locks: Group binding is done during query/load context setup before task publication. The tracker parent is loaded atomically for diagnostics. Reservation checks and accounting traverse task, group, then process; successful descendants roll back when an ancestor rejects a reservation. The thread manager uses its attached group for reservation and shrink. Group error formatting takes a shared lock, and the called formatter does not reacquire that lock. I found no concrete production lock-order or parent-rebinding defect.
  • Lifecycle and ownership: The group reference in the tracker has no ownership cycle through the group's weak resource-context entries. Temporary limiter switches use the attached group without changing a shared tracker's parent. Existing reservation-token release paths retain their captured group. No new static initialization dependency was identified.
  • Configuration and compatibility: No new configuration, persisted data, transaction path, FE/BE variable, wire or storage format, or rolling-upgrade protocol is changed. Existing workload-group/query limit updates remain the source of the displayed effective task limit. The earlier macOS libc++ P1 inline comment (id 4213793496) is fixed at this head: memory_limit.h now uses Doris atomic_shared_ptr, whose header provides the USE_LIBCPP implementation. It is not an active blocker. The older design discussion (id 4128931042) is addressed by the parent hierarchy and is not a still-applicable P0/P1 finding. BE binary/glibc compatibility was not tested because builds were prohibited.
  • Parallel paths, conditions, and performance: I checked ordinary allocator checks, reserve-enabled queries, disabled task allocation checks, write-memory sibling trackers, workload-group and process-only checker masks, rollback, and temporary tracker attachment. All eight reservation checker masks were checked statically against the previous accounting contract. Ordinary allocation retains its inline task/process checks; parent traversal and detailed formatting are outside its success path. No concrete performance regression was found.
  • Observability and data safety: The changed error supplies task, group, and process context. The change does not alter transaction visibility, persistence, data writes, or FE/BE protocol. No new data-correctness or memory-accounting bug was substantiated.

Verification was static only as required by the review prompt. I did not build Doris, run tests, or edit product source.

@linrrzqqq

Copy link
Copy Markdown
Collaborator Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29158 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ff3c1ddd60b9f82a8d7669f0e7399d4b70eb0492, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17621	5599	5513	5513
q2	2107	324	270	270
q3	10242	1408	847	847
q4	4680	471	351	351
q5	7437	799	537	537
q6	193	187	161	161
q7	773	803	630	630
q8	9261	1377	1051	1051
q9	5717	4440	4428	4428
q10	6813	1318	1022	1022
q11	436	248	241	241
q12	627	403	292	292
q13	18167	3088	2384	2384
q14	276	264	248	248
q15	q16	739	719	665	665
q17	1180	733	589	589
q18	7081	6270	6227	6227
q19	1118	959	625	625
q20	410	335	234	234
q21	5845	2554	2606	2554
q22	395	324	289	289
Total cold run time: 101118 ms
Total hot run time: 29158 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	6421	6443	6543	6443
q2	733	594	578	578
q3	5604	5713	5413	5413
q4	2050	2186	1575	1575
q5	5746	5776	5779	5776
q6	254	199	158	158
q7	2204	2045	1811	1811
q8	3135	2754	2701	2701
q9	8184	7708	7740	7708
q10	3671	3601	3234	3234
q11	594	401	396	396
q12	665	708	507	507
q13	2723	3065	2390	2390
q14	283	282	255	255
q15	q16	666	667	625	625
q17	7796	7029	6855	6855
q18	13098	12213	13112	12213
q19	911	792	779	779
q20	2207	2166	1942	1942
q21	5694	4655	4944	4655
q22	488	460	409	409
Total cold run time: 73127 ms
Total hot run time: 66423 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 152204 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ff3c1ddd60b9f82a8d7669f0e7399d4b70eb0492, data reload: false

query5	4305	576	442	442
query6	415	202	189	189
query7	4806	437	220	220
query8	318	189	167	167
query9	8769	4021	4026	4021
query10	473	309	252	252
query11	6017	3409	3185	3185
query12	147	85	83	83
query13	1257	438	337	337
query14	6546	4862	4518	4518
query14_1	4236	4299	4271	4271
query15	207	204	178	178
query16	946	478	445	445
query17	898	635	517	517
query18	2404	405	306	306
query19	188	165	126	126
query20	80	79	78	78
query21	210	140	113	113
query22	12967	13048	12914	12914
query23	13271	12633	12017	12017
query23_1	12233	12294	12141	12141
query24	7093	833	452	452
query24_1	459	467	477	467
query25	505	405	343	343
query26	1270	253	134	134
query27	2779	433	268	268
query28	4663	1912	1914	1912
query29	1527	583	432	432
query30	297	226	184	184
query31	920	774	658	658
query32	135	91	84	84
query33	491	296	239	239
query34	1015	834	499	499
query35	750	768	658	658
query36	836	802	726	726
query37	126	108	94	94
query38	1799	1733	1675	1675
query39	707	752	682	682
query39_1	672	654	672	654
query40	231	121	103	103
query41	78	74	71	71
query42	94	83	83	83
query43	371	392	332	332
query44	1340	696	701	696
query45	187	183	168	168
query46	836	955	567	567
query47	2989	3067	2957	2957
query48	293	307	221	221
query49	577	401	312	312
query50	703	286	218	218
query51	10184	10362	10458	10362
query52	82	82	77	77
query53	192	209	151	151
query54	254	198	202	198
query55	73	69	67	67
query56	239	224	218	218
query57	1624	1583	1583	1583
query58	295	273	258	258
query59	2386	2392	2140	2140
query60	300	250	232	232
query61	178	173	169	169
query62	401	346	294	294
query63	205	170	175	170
query64	2891	1097	911	911
query65	3447	3381	3409	3381
query66	1785	422	292	292
query67	20319	20061	20085	20061
query68	3052	1001	603	603
query69	381	289	243	243
query70	912	845	846	845
query71	296	223	252	223
query72	2727	2565	2257	2257
query73	517	529	295	295
query74	4567	4498	4296	4296
query75	2339	2319	1950	1950
query76	2291	1010	617	617
query77	358	402	290	290
query78	9102	8986	8483	8483
query79	972	847	515	515
query80	1235	433	331	331
query81	585	326	277	277
query82	610	140	116	116
query83	301	209	193	193
query84	328	129	103	103
query85	858	419	372	372
query86	406	240	226	226
query87	2053	1976	1845	1845
query88	3611	2693	2667	2667
query89	346	297	262	262
query90	1940	190	184	184
query91	174	150	122	122
query92	96	88	79	79
query93	1036	980	567	567
query94	639	314	280	280
query95	577	370	389	370
query96	671	520	231	231
query97	2474	2437	2316	2316
query98	160	148	139	139
query99	778	771	641	641
Total cold run time: 235460 ms
Total hot run time: 152204 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.53 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ff3c1ddd60b9f82a8d7669f0e7399d4b70eb0492, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.05	0.05
query3	0.26	0.12	0.12
query4	1.62	0.13	0.12
query5	0.27	0.25	0.24
query6	1.21	0.68	0.68
query7	0.04	0.00	0.01
query8	0.05	0.04	0.03
query9	0.41	0.36	0.36
query10	0.57	0.56	0.57
query11	0.20	0.14	0.14
query12	0.18	0.14	0.15
query13	0.50	0.48	0.48
query14	0.98	0.98	0.98
query15	0.65	0.61	0.63
query16	0.34	0.35	0.35
query17	1.14	1.14	1.13
query18	0.23	0.21	0.21
query19	2.09	1.92	2.01
query20	0.02	0.02	0.01
query21	15.49	0.29	0.15
query22	4.44	0.05	0.04
query23	16.24	0.32	0.13
query24	3.23	0.55	0.47
query25	0.10	0.07	0.05
query26	1.36	0.23	0.15
query27	0.04	0.03	0.04
query28	2.22	0.60	0.39
query29	12.50	4.14	3.29
query30	0.28	0.14	0.13
query31	2.77	0.57	0.33
query32	3.22	0.62	0.53
query33	3.19	3.29	3.21
query34	15.53	4.01	3.39
query35	3.34	3.30	3.32
query36	0.56	0.45	0.45
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.16	0.14	0.15
query41	0.09	0.03	0.02
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 95.89 s
Total hot run time: 24.53 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 100.00% (10/10) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 65.10% (30825/47350)
Line Coverage 49.89% (323710/648851)
Region Coverage 45.38% (261696/576700)
Branch Coverage 47.07% (122816/260945)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (10/10) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.51% (35036/45795)
Line Coverage 61.72% (395786/641272)
Region Coverage 58.16% (333916/574149)
Branch Coverage 59.05% (153608/260138)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (10/10) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.51% (35037/45795)
Line Coverage 61.72% (395824/641272)
Region Coverage 58.16% (333934/574149)
Branch Coverage 59.05% (153618/260138)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (10/10) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.66% (35106/45795)
Line Coverage 61.87% (396784/641272)
Region Coverage 58.35% (335031/574149)
Branch Coverage 59.21% (154036/260138)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants