Repository navigation
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
| int64_t workload_group_cap = slot_memory_policy == TWgSlotMemoryPolicy::NONE | ||
| ? wg_mem_limit | ||
| : query_weighted_mem_limit; | ||
| resource_ctx->memory_context()->set_mem_limit_from_workload_group(effective_limit, |
There was a problem hiding this comment.
你这么改是不行的。
我们现在调用tracker_limit_exceeded_str 这个方法的时候,实际只检查了query 的memlimit,并没有检查workload group的memlimit。
我们workload group 的memlimit 实际是异步生效的,通过后台线程定期刷新来更新query 的memlimit。
你现在的改法把workload group的limit 传递进去,那么后续比如还想打印process 的,就会传递茫茫多的参数。
这块可能的问题是,我们的memtracker 不是一个树状结构,比如 query memtracker----parent ---> workload group memtracker --parent--> process memtracker。
### What problem does this PR solve? Issue Number: None Related PR: apache#68589 Problem Summary: Task, workload group and process limits use separate check and reservation paths, making their relationships and limiting budgets hard to trace. Add a MemoryLimit hierarchy with selected-node checks, recursive reservations and rollback, and ancestor diagnostics. Attach query and write trackers to their workload group while temporary tracker switches use the thread's workload group without reparenting shared trackers. Preserve existing memory accounting, check policies and the dedicated process/task fast paths for ordinary allocations. ### Release note Query memory-limit diagnostics include workload group and process ancestors. ### Check List (For Author) - Test: ASAN BE build and 83 unit tests across MemTrackerTest, ThreadMemTrackerMgrTest, AllocatorTest, WorkloadGroupManagerTest, WorkloadSchedPolicyTest and PipelineTaskTest passed. Header hygiene and clang-format v16 passed. Optimized LLVM IR confirms ordinary allocation checks avoid tree dispatch. clang-tidy retains pre-existing diagnostics outside the modified lines. - Behavior changed: Yes, memory-limit diagnostics show ancestor budgets; allocation/reservation thresholds and check policies are preserved. - Does this need documentation: No.
|
/review |
There was a problem hiding this comment.
Complete static review of PR #68589 at be58ba4 (all 16 changed files). Request changes for one P1: the new MemoryLimit parent field bypasses Doris's libc++ compatible atomic shared pointer and breaks the macOS BE build. The existing inline thread 4128931042 about workload-group/query-limit diagnostic design was considered and is not reposted; it is not a separately confirmed P0/P1 blocker. No additional user review focus was supplied.
Critical checkpoints:
- Goal and proof: The changed error path now prints the task, workload-group, and process limit tree for a query/load tracker with a group parent. Added BE unit tests inspect the diagnostic, local checks, reservation masks, rollback, sibling queries, and binding order. They were read only; no test or end-to-end SQL run proves behavior here.
- Scope and clarity: The new hierarchy and reservation refactor are broader than the diagnostic change described by the PR. The new type is gathered by the runtime CMake glob, but its raw atomic shared pointer is a concrete platform compatibility failure (inline P1).
- Concurrency and lifecycle: Worker-thread reservations still account task, attached workload group, then process, with rollback on failure. The parent is bound during query/load context setup; the workload-group registry holds weak resource contexts, and the diagnostic group string takes a shared lock. No new counter imbalance, ownership cycle, or deadlock was substantiated. The group's preexisting admission check remains non-atomic across load and add.
- Configuration, compatibility, and parallel paths: No new config, wire/storage format, transaction, data-write, or FE-to-BE variable is introduced. Existing workload-group limit changes remain asynchronous. Query, load, write-sibling, tracker-only attachment, and temporary-limiter paths were inspected. The libc++ target is broken as noted above; release ELF/glibc compatibility could not be audited because this review forbids builds.
- Conditions, performance, and observability: Task-only allocation checks remain inline; recursive ancestor checks and tree formatting occur in explicit checks, reservations, or error paths. The check-mask and failure rollback paths were traced, with no additional substantiated issue. The error includes the current effective task limit and group/process context; existing discussion about more specific advice remains on the PR.
- Test and style evidence: The new BE tests cover important boundaries but are unexecuted here, and no end-to-end negative case was added.
git diff --checkpassed; the pure-text BE build-hygiene gate passed. This is a static-only review, with no build or runtime validation.
Review completion: two rounds of normal and risk-focused review converged on this single new inline finding, followed by a full changed-file and unresolved-candidate sweep. All other suspicious points were dismissed with code evidence or deduplicated against the existing thread.
### What problem does this PR solve? Issue Number: None Related PR: apache#68589 Problem Summary: Task, workload group and process limits use separate check and reservation paths, making their relationships and limiting budgets hard to trace. Add a MemoryLimit hierarchy with selected-node checks, recursive reservations and rollback, and ancestor diagnostics. Attach query and write trackers to their workload group while temporary tracker switches use the thread's workload group without reparenting shared trackers. Preserve existing memory accounting, check policies and the dedicated process/task fast paths for ordinary allocations. Use the existing atomic_shared_ptr wrapper for parent storage so libc++ builds compile while preserving atomic load/store semantics. ### Release note Query memory-limit diagnostics include workload group and process ancestors. ### Check List (For Author) - Test: ASAN BE build and 85 unit tests across MemTrackerTest, ThreadMemTrackerMgrTest, AllocatorTest, WorkloadGroupManagerTest, WorkloadSchedPolicyTest and PipelineTaskTest passed. Header hygiene and clang-format v16 passed. The MemoryLimit header and implementation passed libc++ compilation checks; the modified header passed clang-tidy. Optimized LLVM IR confirms ordinary allocation checks avoid tree dispatch. clang-tidy retains pre-existing diagnostics outside the modified lines. - Behavior changed: Yes, memory-limit diagnostics show ancestor budgets; allocation/reservation thresholds and check policies are preserved. - Does this need documentation: No.
|
/review |
There was a problem hiding this comment.
Complete static review of apache/doris PR #68589 at head ff3c1dd. I found no new substantiated inline issue. All 16 changed files were covered by the main review, two normal full-review passes, and a separate risk-focused pass; all three passes returned NO_NEW_VALUABLE_FINDINGS in round 1. No candidate remains unresolved. There was no additional user-provided review focus.
Critical checkpoints:
- Goal, scope, and tests: The change attaches the query task tracker to a workload-group/process limit hierarchy so a query allocation error can show its effective task limit and group context. The added query_error_includes_memory_limit_ancestors BE test asserts the group appears in the error and the old exec_mem_limit-only advice is absent. The hierarchy is broader than the PR description, but the changes are cohesive. These are inspected tests, not executed validation.
- Concurrency and locks: Group binding is done during query/load context setup before task publication. The tracker parent is loaded atomically for diagnostics. Reservation checks and accounting traverse task, group, then process; successful descendants roll back when an ancestor rejects a reservation. The thread manager uses its attached group for reservation and shrink. Group error formatting takes a shared lock, and the called formatter does not reacquire that lock. I found no concrete production lock-order or parent-rebinding defect.
- Lifecycle and ownership: The group reference in the tracker has no ownership cycle through the group's weak resource-context entries. Temporary limiter switches use the attached group without changing a shared tracker's parent. Existing reservation-token release paths retain their captured group. No new static initialization dependency was identified.
- Configuration and compatibility: No new configuration, persisted data, transaction path, FE/BE variable, wire or storage format, or rolling-upgrade protocol is changed. Existing workload-group/query limit updates remain the source of the displayed effective task limit. The earlier macOS libc++ P1 inline comment (id 4213793496) is fixed at this head: memory_limit.h now uses Doris atomic_shared_ptr, whose header provides the USE_LIBCPP implementation. It is not an active blocker. The older design discussion (id 4128931042) is addressed by the parent hierarchy and is not a still-applicable P0/P1 finding. BE binary/glibc compatibility was not tested because builds were prohibited.
- Parallel paths, conditions, and performance: I checked ordinary allocator checks, reserve-enabled queries, disabled task allocation checks, write-memory sibling trackers, workload-group and process-only checker masks, rollback, and temporary tracker attachment. All eight reservation checker masks were checked statically against the previous accounting contract. Ordinary allocation retains its inline task/process checks; parent traversal and detailed formatting are outside its success path. No concrete performance regression was found.
- Observability and data safety: The changed error supplies task, group, and process context. The change does not alter transaction visibility, persistence, data writes, or FE/BE protocol. No new data-correctness or memory-accounting bug was substantiated.
Verification was static only as required by the review prompt. I did not build Doris, run tests, or edit product source.
|
run buildall |
TPC-H: Total hot run time: 29158 ms |
TPC-DS: Total hot run time: 152204 ms |
ClickBench: Total hot run time: 24.53 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
When a workload group applies a lower per-query memory limit, the allocation error still suggests raising
exec_mem_limit. Record the workload group query limit alongside the effective tracker limit, then use that information to give the relevant advice in the error message. The existingcheck_limit()decision logic is unchanged.Release note
Query memory limit errors now identify when the workload group query limit is the active constraint and suggest adjusting the workload group limit or reducing query memory use.
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)