Skip to content

Cherry-pick apache/datafusion#24697: use group clustering terminology - #205

Open
Tristan1900 wants to merge 1 commit into
wenqi.mou/cherry-pick/apache-pr-24737-20261111from
wenqi.mou/cherry-pick/apache-pr-24697-20261111
Open

Tristan1900 wants to merge 1 commit into
wenqi.mou/cherry-pick/apache-pr-24737-20261111from
wenqi.mou/cherry-pick/apache-pr-24697-20261111

Conversation

@Tristan1900

@Tristan1900 Tristan1900 commented Nov 11, 2026 •

Copy link
Copy Markdown
  • Backport apache/datafusion#24697: replace aggregate InputOrderMode with GroupClusteringMode and rename the group-completion trackers and streams.
  • Apply the rename to DataFusion 55's aggregation implementations, callers, plan output, and upgrade guide.
  • Depends on #204.

@codecov-commenter

codecov-commenter commented Nov 11, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.80000% with 43 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.23%. Comparing base (c8fd71b) to head (5247030).

Files with missing lines Patch % Lines
...ical-plan/src/aggregates/clustered_final_stream.rs 80.55% 14 Missing ⚠️
datafusion/physical-plan/src/aggregates/mod.rs 81.81% 3 Missing and 11 partials ⚠️
...tafusion/physical-plan/src/aggregates/order/mod.rs 85.00% 2 Missing and 4 partials ⚠️
...hysical-plan/src/aggregates/grouped_hash_stream.rs 81.25% 0 Missing and 3 partials ⚠️
...ggregates/aggregate_hash_table/common_clustered.rs 84.61% 0 Missing and 2 partials ⚠️
...plan/src/aggregates/aggregate_hash_table/common.rs 0.00% 0 Missing and 1 partial ⚠️
...c/aggregates/aggregate_hash_table/partial_table.rs 0.00% 0 Missing and 1 partial ⚠️
...sion/physical-plan/src/aggregates/order/partial.rs 80.00% 0 Missing and 1 partial ⚠️
datafusion/physical-plan/src/recursive_query.rs 0.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@                                Coverage Diff                                 @@
##           wenqi.mou/cherry-pick/apache-pr-24737-20261111     #205      +/-   ##
==================================================================================
- Coverage                                           81.23%   81.23%   -0.01%     
==================================================================================
  Files                                                1111     1111              
  Lines                                              389136   389178      +42     
  Branches                                           389136   389178      +42     
==================================================================================
+ Hits                                               316107   316135      +28     
- Misses                                              54346    54358      +12     
- Partials                                            18683    18685       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Tristan1900
Tristan1900 marked this pull request as ready for review November 12, 2026 18:23
@Tristan1900
Tristan1900 force-pushed the wenqi.mou/cherry-pick/apache-pr-24697-20261111 branch 3 times, most recently from f77118c to 1928de0 Compare November 12, 2026 20:13
@datadog-official

This comment has been minimized.

@Tristan1900
Tristan1900 added this pull request to stack #213 November 12, 2026 20:50
…ache#24697)

## Which issue does this PR relate to?

- Part of apache#24438.
- Builds on the baseline test in apache#24737 (merged).

## Rationale for this change

Aggregate execution currently uses `InputOrderMode` to select execution
paths and recognize completed groups.

A group can be completed incrementally when its rows form one contiguous
run: once the run ends, that group cannot appear again in the same input
partition. Sorted input supplies this guarantee, and unsorted input can
supply it too:

```text
AAABBBCCCC -> sorted; groups can be completed incrementally
CCCAAABBB  -> unsorted; groups can still be completed incrementally
```

This PR describes the input's _**clustering**_ guarantee with
`GroupClusteringMode` and uses clustering terminology throughout the
aggregate execution machinery. Aggregate construction derives the mode
directly from the existing ordering analysis. The follow-up in apache#24497
will also derive it from grouped equivalence properties.

## What changes are included in this PR?

- Replace the aggregate's `InputOrderMode` field with
`GroupClusteringMode::{None, Partial, Full}`:
  - `None`: Groups have no apparent clustering guarantee.
- `Partial(indices)`: rows are contiguous on a subset of grouping
expressions; a change in that tuple completes all groups in the
preceding run.
- `Full`: rows are contiguous on the complete grouping tuple; a change
in that tuple completes the preceding group.
- Rename the stateful `GroupOrdering` tracker and its `Partial` and
`Full` implementations to `GroupClustering`, `GroupClusteringPartial`,
and `GroupClusteringFull`.
- Rename the ordered aggregate streams, shared table, and adjacent-key
group-values implementation to use clustered terminology,
- Preserve actual sort expressions and options through equivalence
properties and input-ordering requirements.
- Display `group_clustering_mode` in aggregate plans and document the
public API migration in the DataFusion 56 upgrade guide.

## Stack

1. [apache#24737 — test: cover unsorted contiguous groups in one
partition](apache#24737) (merged)
2. [apache#24697 — refactor: use group clustering terminology in aggregate
execution](apache#24697) ← **this
PR**
3. [apache#24698 — feat: add Grouped equivalence
properties](apache#24698)
4. [apache#24497 — feat: stream aggregates over grouped
input](apache#24497)

## Are there any user-facing changes?

Yes. This PR changes public Rust APIs:

- `AggregateExec::input_order_mode()` is replaced by
`AggregateExec::group_clustering_mode()`, returning the public
`GroupClusteringMode` enum.
- `AggregateExec::compute_properties` accepts `&GroupClusteringMode` in
place of `&InputOrderMode`.
- The public `GroupOrdering`, `GroupOrderingPartial`, and
`GroupOrderingFull` types in `aggregates::order` are renamed to
`GroupClustering`, `GroupClusteringPartial`, and `GroupClusteringFull`.
- `GroupClustering::try_new` takes `&GroupClusteringMode`;
`new_group_values` takes `&GroupClustering`.

Aggregate plan output uses `group_clustering_mode=Full` and
`group_clustering_mode=Partial(indices)` in place of
`ordering_mode=Sorted` and `ordering_mode=PartiallySorted(indices)`. SQL
results and aggregate execution-path selection retain their existing
behavior.

## Review this layer

[View this PR's
diff](https://github.com/apache/datafusion/pull/24697/files)

(cherry picked from commit d93699c)
@erik-sargent
erik-sargent force-pushed the wenqi.mou/cherry-pick/apache-pr-24697-20261111 branch from 1928de0 to 5247030 Compare November 16, 2026 18:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants