Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
ac76b0d
fix: only apply SingleDistinctToGroupBy rewrite when it pays (closes …
mohitgurav20 Oct 17, 2026
09f9d96
Merge branch 'main' into fix/single-distinct-groupby-memory-usage
mohitgurav20 Oct 17, 2026
bbf4ccb
test: cover rewrite_pays for unsupported distinct handling in SingleD…
mohitgurav20 Oct 17, 2026
222ff4e
feat: add single_distinct_aggregation_prefer_memory optimizer option
mohitgurav20 Oct 19, 2026
cca6cd2
Merge branch 'main' into fix/single-distinct-groupby-memory-usage
mohitgurav20 Oct 19, 2026
f9c72ea
feat(optimizer): refine single distinct rewrite decision tree
mohitgurav20 Oct 19, 2026
3739116
Merge branch 'main' into fix/single-distinct-groupby-memory-usage
mohitgurav20 Oct 19, 2026
df3a448
test(optimizer): add branch coverage tests for is_single_distinct_agg
mohitgurav20 Oct 19, 2026
7f375a3
Merge branch 'main' into fix/single-distinct-groupby-memory-usage
mohitgurav20 Oct 19, 2026
1609f89
fix(docs): escape angle brackets in is_single_distinct_agg doc comment
mohitgurav20 Oct 19, 2026
d3e37ce
Merge branch 'main' into fix/single-distinct-groupby-memory-usage
mohitgurav20 Oct 19, 2026
4587fde
fix(optimizer): preserve count rollup guard and update tpch q16 plan
mohitgurav20 Oct 20, 2026
341df0f
Merge branch 'main' into fix/single-distinct-groupby-memory-usage
mohitgurav20 Oct 20, 2026
51f1775
test(sqllogictest): update plan snapshots for single distinct to grou…
mohitgurav20 Oct 20, 2026
38468c5
Merge branch 'main' into fix/single-distinct-groupby-memory-usage
mohitgurav20 Oct 20, 2026
7e4a93c
Merge branch 'main' into fix/single-distinct-groupby-memory-usage
mohitgurav20 Oct 21, 2026
5d437a3
test(optimizer): eliminate unexercised test branches and add avg comp…
mohitgurav20 Oct 21, 2026
5aa5906
Merge remote-tracking branch 'apache/main' into fix/single-distinct-g…
mohitgurav20 Oct 28, 2026
095b8c7
Merge remote-tracking branch 'apache/main' into fix/single-distinct-g…
mohitgurav20 Oct 28, 2026
2a091fa
ci: trigger mergeability recheck after sync with apache/main
mohitgurav20 Oct 28, 2026
1682661
Merge remote-tracking branch 'apache/main' into fix/single-distinct-g…
mohitgurav20 Oct 29, 2026
03dbe3e
Merge branch 'main' into fix/single-distinct-groupby-memory-usage
mohitgurav20 Oct 30, 2026
051a9d2
fix(optimizer): gate single distinct rewrite with single_distinct_agg…
mohitgurav20 Nov 1, 2026
eb2cd8a
Merge branch 'main' into fix/single-distinct-groupby-memory-usage
mohitgurav20 Nov 1, 2026
8024c82
fix: resolve intra-doc link for DistinctHandling in single_distinct_t…
mohitgurav20 Nov 1, 2026
a077c0c
Merge branch 'main' into fix/single-distinct-groupby-memory-usage
mohitgurav20 Nov 1, 2026
8071b00
Merge remote-tracking branch 'apache/main' into fix/single-distinct-g…
mohitgurav20 Nov 2, 2026
b4f3d08
feat(optimizer): refine prefer_memory guard to avoid regressions on s…
mohitgurav20 Nov 2, 2026
16000fe
Merge remote-tracking branch 'apache/main' into fix/single-distinct-g…
mohitgurav20 Nov 2, 2026
2341fe3
style: format single_distinct_to_groupby.rs
mohitgurav20 Nov 2, 2026
f45925c
docs: fix doc comment placement in single_distinct_to_groupby.rs
mohitgurav20 Nov 2, 2026
93e3746
Merge branch 'main' into fix/single-distinct-groupby-memory-usage
mohitgurav20 Nov 2, 2026
c8df6bc
Merge remote-tracking branch 'apache/main' into fix/single-distinct-g…
mohitgurav20 Nov 3, 2026
24155f6
style: add trailing newline to single_distinct_to_groupby.slt
mohitgurav20 Nov 7, 2026
c413a0e
Merge remote-tracking branch 'apache/main' into fix/single-distinct-g…
mohitgurav20 Nov 8, 2026
defd182
Merge remote-tracking branch 'apache/main' into fix/single-distinct-g…
mohitgurav20 Nov 9, 2026
1f7c26b
fix: address review feedback on float distinct semantics and document…
mohitgurav20 Nov 9, 2026
1b4e42b
refactor(optimizer): consolidate single distinct rewrite settings int…
mohitgurav20 Nov 11, 2026
678fe74
fix: exclude Duration from native distinct and clarify group size mem…
mohitgurav20 Nov 12, 2026
8f12668
docs(config): clarify single_distinct_prefer_native requirements and …
mohitgurav20 Nov 14, 2026
10cc8df
refactor: address alamb review feedback on doc phrasing and slt tests
mohitgurav20 Nov 14, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 7 additions & 9 deletions datafusion/common/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1940,16 +1940,14 @@ config_namespace! {
/// projections or aliases.
pub enable_unions_to_filter: bool, default = false

/// Prefer native grouped DISTINCT aggregation when target_partitions is 1
/// and every aggregate explicitly supports a specialized GroupsAccumulator
/// for its DISTINCT argument types. Defaults to false, preserving the
/// two-phase GROUP BY rewrite.
/// When `true`, queries with a single DISTINCT aggregate will take less
/// memory but may also run more slowly, by skipping the two-phase `GROUP BY`
/// rewrite and executing with native accumulators.
///
/// The native plan can avoid materializing distinct (group, value) pairs,
/// but can be slower for many groups with duplicate values. It must fit
/// each group's distinct values in memory, so queries that succeed by
/// spilling with the rewrite may fail with this option enabled. One target
/// partition can still have multiple input partitions and merge native states.
/// Only applies when `target_partitions` is 1 and all DISTINCT arguments are
/// fixed-width, non-float types. See
/// [`SingleDistinctToGroupBy`](https://docs.rs/datafusion-optimizer/latest/datafusion_optimizer/single_distinct_to_groupby/struct.SingleDistinctToGroupBy.html)
/// for details.
pub single_distinct_prefer_native: bool, default = false
}
}
Expand Down
97 changes: 84 additions & 13 deletions datafusion/optimizer/src/single_distinct_to_groupby.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ use std::sync::Arc;

use crate::optimizer::ApplyOrder;
use crate::{OptimizerConfig, OptimizerRule};
use arrow::datatypes::DataType;

use datafusion_common::{
DFSchema, DataFusionError, HashSet, Result, assert_or_internal_err,
Expand All @@ -17,7 +18,7 @@ use datafusion_expr::builder::project;
use datafusion_expr::expr::AggregateFunctionParams;
use datafusion_expr::expr_schema::ExprSchemable;
use datafusion_expr::{
AggregateUDF, Expr, cast, col,
AggregateUDF, DistinctHandling, Expr, cast, col,
expr::AggregateFunction,
lit,
logical_plan::{Aggregate, LogicalPlan},
Expand Down Expand Up @@ -68,9 +69,20 @@ use datafusion_expr::{
/// `sum` of no rows is NULL.
///
/// See `rewrite_pays_for_count` for when the `count` is allowed.
///
/// # Memory vs Performance Tradeoffs (`single_distinct_prefer_native`)
///
/// With `single_distinct_prefer_native` enabled and one target partition, grouped
/// all-DISTINCT aggregates stay native if every function explicitly supports a
/// specialized `GroupsAccumulator` for its argument types.
/// all-DISTINCT aggregates stay native when every DISTINCT argument is a
/// fixed-width, non-float type (such as integers, decimals, dates).
///
/// Skipping the two-phase rewrite avoids materializing intermediate
/// `(group, distinct value)` pairs, lowering peak memory when there are few
/// groups with many distinct values each. With many small groups it can use
/// more memory and run slower, because functions without a specialized DISTINCT
/// `GroupsAccumulator` maintain one accumulator per group. Queries without
/// `GROUP BY`, multi-partition queries, or queries with float/variable-length
/// arguments continue to use the two-phase rewrite.
#[derive(Default, Debug)]
pub struct SingleDistinctToGroupBy {}

Expand Down Expand Up @@ -110,7 +122,49 @@ impl CountRollup {
}
}

/// Check whether all aggregate exprs are distinct on a single field.
/// Returns `true` for fixed-width primitive types where single-phase distinct
/// accumulation avoids heap allocation churn **and** where bit-identity equals
/// value-equality, so that skipping the GROUP BY rewrite cannot silently change
/// query results.
///
/// Float types are deliberately excluded: IEEE 754 defines `-0.0 == 0.0`, so a
/// `GROUP BY` on floats merges those two bit-patterns into one group. The
/// native distinct accumulators hash raw bits and therefore count them
/// separately. Keeping floats in this list would let `single_distinct_prefer_native`
/// skip the rewrite for a float column, producing a different answer than the
/// non-rewritten path — a silent correctness regression.
///
/// `Duration` is also excluded: `avg` has no native DISTINCT accumulator for duration
/// types, so skipping the rewrite would cause a runtime error.
fn is_fixed_width_type(dt: &DataType) -> bool {
matches!(
dt,
DataType::Boolean
| DataType::Int8
| DataType::Int16
| DataType::Int32
| DataType::Int64
| DataType::UInt8
| DataType::UInt16
| DataType::UInt32
| DataType::UInt64
| DataType::Date32
| DataType::Date64
| DataType::Time32(_)
| DataType::Time64(_)
| DataType::Timestamp(_, _)
| DataType::Interval(_)
| DataType::Decimal128(_, _)
| DataType::Decimal256(_, _)
)
}

/// Check whether all aggregate exprs are eligible for the single-distinct-to-group-by rewrite.
///
/// When `use_native_grouped_distinct` is set, all-DISTINCT queries on fixed-width,
/// non-float types skip the rewrite and run natively to reduce intermediate memory.
/// All other queries (un-grouped, multi-partition, mixed distinct, unsupported UDAFs,
/// floats, or variable-width types) proceed with the rewrite.
fn is_single_distinct_agg(
aggr_expr: &[Expr],
input_schema: &DFSchema,
Expand Down Expand Up @@ -158,24 +212,41 @@ fn is_single_distinct_agg(
if aggregate_count != aggr_expr.len() || fields_set.len() != 1 {
return Ok(false);
}
// With one target partition, a native GroupsAccumulator avoids building an
// extra aggregate with a row per (group, distinct value) pair. With multiple
// Aggregates with DistinctHandling::Unsupported (e.g. approx_median) cannot
// run natively with DISTINCT; they must always be rewritten.
if distinct_aggs
.iter()
.any(|(func, _)| func.distinct_handling() == DistinctHandling::Unsupported)
{
return Ok(true);
}
Comment on lines +215 to +222

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the setting off, this return also changes main's behavior: SELECT g, count(*), stddev(DISTINCT x) FROM t GROUP BY g fails on main with STDDEV_POP(DISTINCT) aggregations are not available and returns the right rows here. That's a fix, but no test covers it at the default, and the description says the default is unchanged. Could you add this test before the SET … = true line in single_distinct_to_groupby.slt and mention the fix in the description?

# stddev cannot run DISTINCT itself, so it is rewritten even next to count(*).
query IIR rowsort
SELECT g, count(*), stddev(DISTINCT x) FROM t_prefer_mem GROUP BY g;
----
1 2 7.071067811865
2 1 NULL


// With one target partition, native accumulation avoids building an extra
// aggregate with a row per (group, distinct value) pair. With multiple
// partitions, the rewrite also distributes deduplication by both keys;
// keeping DISTINCT instead repartitions lists of values by group and merges
// them in the final accumulator. That can concentrate work on a few groups.
// Even one target partition can have multiple input partitions (e.g. UNION
// ALL or an existing partitioned table), requiring Partial/Final state merging.
// This is opt-in: the rule cannot see memory limits or cost the input, and
// native DISTINCT must fit each group's distinct values in memory.
if use_native_grouped_distinct
&& distinct_aggs.len() == aggregate_count
// An unknown answer must preserve the existing rewrite.
&& distinct_groups_support(&distinct_aggs, input_schema)?
//
// The rewrite is skipped only when every aggregate is a DISTINCT aggregate
// over fixed-width primitive types (integers, decimals, dates, etc.) where
// bit-identity equals value-equality. Floats are excluded to avoid -0.0 vs
// 0.0 IEEE 754 discrepancy, and variable-length types (strings, binary) are
// excluded to prevent heap allocation churn.
if use_native_grouped_distinct && distinct_aggs.len() == aggregate_count {
let arg_types = distinct_aggs
.iter()
.all(|support| *support == Some(true))
{
return Ok(false);
.flat_map(|(_, args)| args.iter())
.map(|arg| arg.get_type(input_schema))
.collect::<Result<Vec<_>>>()?;
if arg_types.iter().all(is_fixed_width_type) {
return Ok(false);
}
}

if has_count_rollup && !rewrite_pays_for_count(&distinct_aggs, input_schema)? {
return Ok(false);
}
Expand Down
19 changes: 12 additions & 7 deletions datafusion/sqllogictest/test_files/grouped_count_distinct.slt
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@
# SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership
# SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements

# With the opt-in enabled and one target partition, grouped integer DISTINCT
# counts keep their specialized accumulator. Multiple target partitions retain
# With the opt-in enabled and one target partition, grouped all-DISTINCT aggregates
# with fixed-width, non-float arguments stay native. Multiple target partitions retain
# distributed deduplication.
# String and floating-point DISTINCT arguments still use the two-phase rewrite.

Expand Down Expand Up @@ -80,15 +80,20 @@ logical_plan
03)----Aggregate: groupBy=[[numeric_counts.g, numeric_counts.x AS alias1]], aggr=[[sum(CAST(numeric_counts.x AS Int64)) AS alias2]]
04)------TableScan: numeric_counts projection=[g, x]

# `sum` does not report specialized DISTINCT support, so it keeps the rewrite.
# `sum` with integer DISTINCT argument stays native.
query TT
EXPLAIN SELECT g, sum(DISTINCT x) FROM numeric_counts GROUP BY g;
----
logical_plan
01)Projection: numeric_counts.g, sum(alias1) AS sum(DISTINCT numeric_counts.x)
02)--Aggregate: groupBy=[[numeric_counts.g]], aggr=[[sum(alias1)]]
03)----Aggregate: groupBy=[[numeric_counts.g, CAST(numeric_counts.x AS Int64) AS alias1]], aggr=[[]]
04)------TableScan: numeric_counts projection=[g, x]
01)Aggregate: groupBy=[[numeric_counts.g]], aggr=[[sum(DISTINCT CAST(numeric_counts.x AS Int64))]]
02)--TableScan: numeric_counts projection=[g, x]

query II rowsort
SELECT g, sum(DISTINCT x) FROM numeric_counts GROUP BY g;
----
1 3
2 NULL
3 3

# The global aggregate does not use a GroupsAccumulator.
query TT
Expand Down
2 changes: 1 addition & 1 deletion datafusion/sqllogictest/test_files/information_schema.slt
Original file line number Diff line number Diff line change
Expand Up @@ -505,7 +505,7 @@ datafusion.optimizer.repartition_file_scans true When set to `true`, datasource
datafusion.optimizer.repartition_joins true Should DataFusion repartition data using the join keys to execute joins in parallel using the provided `target_partitions` level
datafusion.optimizer.repartition_sorts true Should DataFusion execute sorts in a per-partition fashion and merge afterwards instead of coalescing first and sorting globally. With this flag is enabled, plans in the form below ```text "SortExec: [a@0 ASC]", " CoalescePartitionsExec", " RepartitionExec: partitioning=RoundRobinBatch(8), input_partitions=1", ``` would turn into the plan below which performs better in multithreaded environments ```text "SortPreservingMergeExec: [a@0 ASC]", " SortExec: [a@0 ASC]", " RepartitionExec: partitioning=RoundRobinBatch(8), input_partitions=1", ```
datafusion.optimizer.repartition_windows true Should DataFusion repartition data using the partitions keys to execute window functions in parallel using the provided `target_partitions` level
datafusion.optimizer.single_distinct_prefer_native false Prefer native grouped DISTINCT aggregation when target_partitions is 1 and every aggregate explicitly supports a specialized GroupsAccumulator for its DISTINCT argument types. Defaults to false, preserving the two-phase GROUP BY rewrite. The native plan can avoid materializing distinct (group, value) pairs, but can be slower for many groups with duplicate values. It must fit each group's distinct values in memory, so queries that succeed by spilling with the rewrite may fail with this option enabled. One target partition can still have multiple input partitions and merge native states.
datafusion.optimizer.single_distinct_prefer_native false When `true`, queries with a single DISTINCT aggregate will take less memory but may also run more slowly, by skipping the two-phase `GROUP BY` rewrite and executing with native accumulators. Only applies when `target_partitions` is 1 and all DISTINCT arguments are fixed-width, non-float types. See [`SingleDistinctToGroupBy`](https://docs.rs/datafusion-optimizer/latest/datafusion_optimizer/single_distinct_to_groupby/struct.SingleDistinctToGroupBy.html) for details.
datafusion.optimizer.skip_failed_rules false When set to true, the logical plan optimizer will produce warning messages if any optimization rules produce errors and then proceed to the next rule. When set to false, any rules that produce errors will cause the query to fail
datafusion.optimizer.subset_repartition_threshold 4 Partition count threshold for subset satisfaction optimization. When the current partition count is >= this threshold, DataFusion will skip repartitioning if the required partitioning expression is a subset of the current partition expression such as Hash(a) satisfies Hash(a, b). When the current partition count is < this threshold, DataFusion will repartition to increase parallelism even when subset satisfaction applies. Set to 0 to always repartition (disable subset satisfaction optimization). Set to a high value to always use subset satisfaction. Example (subset_repartition_threshold = 4): ```text Hash([a]) satisfies Hash([a, b]) because (Hash([a, b]) is subset of Hash([a]) If current partitions (3) < threshold (4), repartition: AggregateExec: mode=FinalPartitioned, gby=[a, b], aggr=[SUM(x)] RepartitionExec: partitioning=Hash([a, b], 8), input_partitions=3 AggregateExec: mode=Partial, gby=[a, b], aggr=[SUM(x)] DataSourceExec: file_groups={...}, output_partitioning=Hash([a], 3) If current partitions (8) >= threshold (4), use subset satisfaction: AggregateExec: mode=SinglePartitioned, gby=[a, b], aggr=[SUM(x)] DataSourceExec: file_groups={...}, output_partitioning=Hash([a], 8) ```
datafusion.optimizer.top_down_join_key_reordering true When set to true, the physical plan optimizer will run a top down process to reorder the join keys
Expand Down
Loading
Loading