Skip to content

Cherry-pick apache/datafusion#24737: cover unsorted contiguous groups - #204

Open
Tristan1900 wants to merge 1 commit into
branch-55from
wenqi.mou/cherry-pick/apache-pr-24737-20261111
Open

Tristan1900 wants to merge 1 commit into
branch-55from
wenqi.mou/cherry-pick/apache-pr-24737-20261111

Conversation

@Tristan1900

@Tristan1900 Tristan1900 commented Nov 11, 2026 •

Copy link
Copy Markdown
  • Backport apache/datafusion#24737, covering contiguous groups whose key order resets within one input partition.
  • Check grouped sums and the existing hash-aggregation path before enabling incremental emission.

@codecov-commenter

codecov-commenter commented Nov 11, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 81.03448% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.23%. Comparing base (e7af2a2) to head (c8fd71b).

Files with missing lines Patch % Lines
datafusion/physical-plan/src/aggregates/mod.rs 81.03% 3 Missing and 8 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           branch-55     #204   +/-   ##
==========================================
  Coverage      81.22%   81.23%           
==========================================
  Files           1111     1111           
  Lines         389078   389136   +58     
  Branches      389078   389136   +58     
==========================================
+ Hits          316046   316107   +61     
+ Misses         54358    54346   -12     
- Partials       18674    18683    +9     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Tristan1900
Tristan1900 marked this pull request as ready for review November 12, 2026 18:23
@Tristan1900
Tristan1900 force-pushed the wenqi.mou/cherry-pick/apache-pr-24737-20261111 branch from 55db7eb to e9a11d2 Compare November 12, 2026 18:36
@Tristan1900
Tristan1900 force-pushed the wenqi.mou/cherry-pick/apache-pr-24737-20261111 branch 2 times, most recently from 1a24291 to 65e53a7 Compare November 12, 2026 20:13
@Tristan1900
Tristan1900 changed the base branch from wenqi.mou/cherry-pick/apache-pr-25019-20261111 to branch-55 November 12, 2026 20:13
@datadog-prod-us1-3

This comment has been minimized.

@Tristan1900
Tristan1900 added this pull request to stack #213 November 12, 2026 20:50
## Which issue does this PR relate to?

- Part of apache#24438.

## Rationale for this change

A source can concatenate several sorted logical runs into one DataFusion
output partition. The resulting stream may be globally unsorted while
every distinct `(key, time_bin)` tuple still occupies one contiguous
range.

This PR records how aggregate planning handles that layout before
grouped input properties are available. It provides the behavioral
baseline for the remaining PRs in the stack.

## What changes are included in this PR?

- Add a single-partition `TestMemoryExec` fixture containing two sorted
logical runs whose `(key, time_bin)` order resets at the record-batch
boundary.
- Aggregate by the complete `(key, time_bin)` tuple and verify the
result.
- Assert that planning selects `InputOrderMode::Linear`,
`EmissionType::Final`, and `SingleHashAggregateStream`.

## Stack

1. [apache#24737 — test: cover unsorted contiguous groups in one
partition](apache#24737) ← **this
PR**
2. [apache#24697 — refactor: separate aggregate group completion from input
ordering](apache#24697)
3. [apache#24698 — feat: add grouped equivalence
properties](apache#24698)
4. [apache#24497 — feat: stream aggregates over grouped
input](apache#24497)

## Are these changes tested?

The new aggregate test executes the single-partition input and
snapshot-checks all four grouped sums.

## Are there any user-facing changes?

No.

## Review this layer

[View only this PR
layer](apache@36969e7)

(cherry picked from commit 59d74ca)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants