Repository navigation
[fix](be) Apply HAVING before the distinct streaming limit - #70512
Conversation
### Background Distinct streaming aggregation emits new grouping keys as input arrives. BE can use this operator for GROUP BY without aggregate functions. A final aggregate can also contain HAVING and LIMIT. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Bugfix Problem Summary: A final distinct streaming aggregate can discard valid groups when it has HAVING and LIMIT. For example, HAVING rejects group 1 and accepts group 2, but LIMIT 1 stops input after group 1. Apply the final limit after HAVING so the query returns group 2. ### Details - **Frequency:** Queries can lose rows when an early group fails HAVING and consumes the limit before filtering. - **Root cause:** push() truncated the aggregate block and set _reach_limit before pull() evaluated HAVING. - **Why existing tests missed it:** The existing operator tests covered LIMIT and local preaggregation without a final HAVING filter. - **Fix:** Skip early truncation when conjuncts are present. Reuse reached_limit() after the final HAVING filter. Preserve output accounting for local preaggregation and its handling of rows that bypass deduplication. - **Other paths and branches:** Ordinary aggregation and streaming aggregation already filter before applying the final limit. No release branch is changed. - **Behavior change:** LIMIT counts only groups that pass HAVING in the final distinct streaming aggregate. ### Testing - Before the fix, all four new BE tests failed. The new SQL suite also failed: it expected 2 and received no row. - Built ASAN BE with `DISABLE_BE_JAVA_EXTENSIONS=ON DISABLE_BE_CDC_CLIENT=ON ./build.sh --be -j24`. The build and its GLIBC_2.17 symbol check passed. - Ran `./run-be-ut.sh --run --filter='*DistinctStreamingAgg*' -j24`. All 15 tests passed. - Ran `./run-regression-test.sh --run -d query_p0/aggregate,query_p0/set_operations,correctness_p0 -s distinct_streaming_having_limit,distinct_streaming_repeated_keys,test_distinct_streaming_agg_local_shuffle`. All three suites passed on the local ASAN BE. - The new SQL suite uses two rows. It checks one final node for HAVING and LIMIT, then checks the result with distinct streaming on and off at batch sizes 1 and 1024. The expected output was generated by the regression script through the ordinary aggregate. - Direct SQL checks returned 2 for both batch sizes. Profiles confirmed the distinct streaming implementation. - clang-format v16, check-format, build structure checks, and clang-tidy passed for the changed scope. clang-tidy used the BE and UT compile commands with the compiler wrapper's toolchain paths. ### Release note Fix missing result rows for GROUP BY queries with HAVING and LIMIT when the final aggregate uses distinct streaming aggregation. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: OpenAI Codex - Generated: Code, tests, comments, and this description - Human review: Pending author review
|
/review |
There was a problem hiding this comment.
Reviewed all four changed files at dc0d2dc and traced FE planning, BE operator selection, task-local execution, buffering, filtering, limits and cleanup. No actionable new findings. No existing inline findings require confirmation. The review converged after 1 round: both complete-scope reviewers and the separate buffered-output risk reviewer returned NO_NEW_VALUABLE_FINDINGS; every risk was independently checked and dismissed with evidence.
Critical checkpoint conclusions:
- Goal and correctness: final distinct aggregation now applies HAVING before LIMIT and counts accepted groups once. Rejected groups no longer consume the allowance. The added BE and SQL cases directly target this failure.
- Scope and reuse: the production change is focused on push/pull and reuses reached_limit(), consistent with the existing aggregation paths.
- Concurrency and locking: changed counters, predicates and buffers are task-local; predicate contexts are cloned and scheduler execution is mutually excluded. No threads, locks, lock order or dependency behavior changed.
- Lifecycle and initialization: no new owners or statics. Cached groups drain across child EOF, and existing close clears remaining output/cache/arena after early termination.
- Configuration: no new settings or dynamic-configuration behavior; tests use existing switches.
- Compatibility: no changed function signatures, protocol or storage formats. No new libc API/dependency is introduced; binary compatibility was not executed or audited.
- Thrift/protobuf: no definition changes; cross-branch field-ID checks are inapplicable.
- Parallel paths: ordinary aggregation suppresses early limiting with HAVING and filters before final limiting; streaming aggregation also filters before its final limit. Local distinct preaggregation keeps its prior counting/pass-through behavior; final distinct remains outside the spill path.
- Conditional checks: the added conjunct guard and final/local split have concrete comments explaining rejected groups and duplicate pass-through rows. No speculative error checks were added.
- Test coverage: four BE cases cover rejected-first-group behavior in one block and across child fetches at batch sizes 1 and 1024, including output/count/EOS assertions. SQL compares ordinary and distinct aggregation at both sizes and checks HAVING and LIMIT on the final node. Existing tests cover plain limits, preaggregation/cache, memory fallback and repeated/nullable keys. No material static coverage gap was identified.
- Expected results: all four new output values of 2 are correct; RAND() is in [0,1), so key 1 fails and key 2 passes. Ordering and repeated output tags match the suite. Output-generation provenance was reported by the author and was not independently reproduced.
- Observability: final output now uses the common processed-row accounting in reached_limit(); existing output/hash/input/memory counters remain available. No additional logging or metrics are needed for this correction.
- Transactions and persistence: no transaction, EditLog, failover, version-visibility or delete-bitmap behavior is changed.
- Data writes and atomicity: production changes affect query-local aggregation only; no data mutation path changes.
- FE/BE variables: no new transmitted state. A retained physical HAVING filter also prevents the translator's separate inner-preaggregation LIMIT pushdown in the regression shape.
- Performance: no new hot-loop copies, repeated scans or allocations. Reading beyond rejected groups is required to return the correct result; limit checks and counter updates are constant-time.
- Other invariants: nullable conversion still precedes HAVING, predicate failures propagate before counting, and existing memory/COW ownership remains unchanged. The final sweep found no additional actionable issue.
User focus: no additional focus points were supplied; the whole PR was reviewed. No unresolved static candidates or assigned coverage remain. This execution was static inspection only: builds, tests, runtime plans/profiles, style tools and binary checks were not run, and the author's execution claims were not independently reproduced.
Review converged after 1 round(s).
|
run buildall |
1 similar comment
|
run buildall |
|
PR approved by anyone and no changes requested. |
|
PR approved by at least one committer and no changes requested. |
Background
Distinct streaming aggregation emits new grouping keys as input arrives. BE can use this operator for GROUP BY without aggregate functions. A final aggregate can also contain HAVING and LIMIT.
What problem does this PR solve?
Issue Number: None
Related PR: None
PR Type: Bugfix
Problem Summary:
A final distinct streaming aggregate can discard valid groups when it has HAVING and LIMIT. For example, HAVING rejects group 1 and accepts group 2, but LIMIT 1 stops input after group 1. Apply the final limit after HAVING so the query returns group 2.
Details
Testing
DISABLE_BE_JAVA_EXTENSIONS=ON DISABLE_BE_CDC_CLIENT=ON ./build.sh --be -j24. The build and its GLIBC_2.17 symbol check passed../run-be-ut.sh --run --filter='*DistinctStreamingAgg*' -j24. All 15 tests passed../run-regression-test.sh --run -d query_p0/aggregate,query_p0/set_operations,correctness_p0 -s distinct_streaming_having_limit,distinct_streaming_repeated_keys,test_distinct_streaming_agg_local_shuffle. All three suites passed on the local ASAN BE.Release note
Fix missing result rows for GROUP BY queries with HAVING and LIMIT when the final aggregate uses distinct streaming aggregation.
Check List (For Author)
AI Disclosure
Background
What problem does this PR solve?
Issue Number:
Related PR:
PR Type:
Problem Summary:
Details
Testing
Notes for Reviewers
Release note
Check List (For Author)
AI Disclosure