Repository navigation
Description
Neither producer of multi-set aggregates shows what the column holds: Isthmus's emit leaves the index out, and DuckDB's root names one column too few for the readers I ran to accept the plan (duckdb-substrait-extension 287).
Describe the bug
The output of an AggregateRel with more than one grouping set ends with "an extra i32 column" whose value "will be the zero-based index of the grouping set that yielded the record" (logical_relations.md at v0.102.0). DataFusion's consumer maps that column to its own __grouping_id, so executing the plan puts the grouping-id bitmask in it rather than the set's index. For the two sets below that bitmask is a UInt8. #23468 moved the column to the position Substrait gives it and left its contents as they were.
The producer makes the same identification when it writes a plan: it maps DataFusion's [groups, grouping_id, measures] onto Substrait's [groups, measures, grouping_id] (producer/rel/aggregate_rel.rs:57-58). A plan passed between DataFusion and an implementation that follows the spec, such as substrait-java, which types the column i32, is therefore read with two meanings for it.
To Reproduce
On main at f8cc67827, this plan has two grouping sets, (a) and (b), over a two-row virtual table:
{"relations": [{"root": {
"names": ["a", "b", "grouping_set"],
"input": {"aggregate": {
"input": {"read": {
"baseSchema": {"names": ["a", "b"], "struct": {"types": [
{"i64": {"nullability": "NULLABILITY_REQUIRED"}},
{"i64": {"nullability": "NULLABILITY_REQUIRED"}}], "nullability": "NULLABILITY_REQUIRED"}},
"virtualTable": {"expressions": [
{"fields": [{"literal": {"i64": "1"}}, {"literal": {"i64": "10"}}]},
{"fields": [{"literal": {"i64": "2"}}, {"literal": {"i64": "20"}}]}]}}},
"groupingExpressions": [
{"selection": {"directReference": {"structField": {"field": 0}}, "rootReference": {}}},
{"selection": {"directReference": {"structField": {"field": 1}}, "rootReference": {}}}],
"groupings": [{"expressionReferences": [0]}, {"expressionReferences": [1]}]}}}}]}Consumed with from_substrait_plan and executed, it returns:
grouping_set: UInt8 (nullable: false)
+---+----+--------------+
| a | b | grouping_set |
+---+----+--------------+
| 2 | | 1 |
| | 20 | 2 |
| 1 | | 1 |
| | 10 | 2 |
+---+----+--------------+
Expected behavior
grouping_set as a required Int32, holding 0 on the two rows from (a) and 1 on the two rows from (b). The bitmask and the index coincide for some lists of sets: for (a, b) followed by (a) both are 0 and then 1. A test needs a list on which they differ, such as (a) followed by (b).
Additional context
First seen in a relation conformance case.
Activity
Mark the grouping-set reason reported
namanjain24-sudo commentedon Sep 20, 2026
take
alamb commentedon Oct 24, 2026
@alexandrefimov -- there is a proposed PR for this here, could you help review?
alamb commentedon Oct 24, 2026
Also, can you please explain what you trying to do?
How is this related to datafusion? Are you trying to run a datafusion plan in duckdb or ishmus? If so can you please share a (small) reproducer of what you tried and what isn't working?
namanjain24-sudo commentedon Oct 24, 2026
(Answering as the author of #25527, the proposed fix — @alexandrefimov filed this issue, so he should correct me if I have his intent wrong.)
As I read it, the problem is entirely inside DataFusion, and nothing here requires running a DataFusion plan in DuckDB or Isthmus. Those two are mentioned only as engines that were checked for a reference plan to compare against.
The spec (logical_relations.md v0.102.0):
DataFusion identifies that column with its own
__grouping_id, which holds a bitmask, not an index. The producer does this explicitly —grouping_set_output_mappingis documented as mapping "Substrait's[groups, measures, grouping_id]" onto DataFusion's[groups, grouping_id, measures]— and the consumer reads it back the same way.Reproducer, DataFusion only. The plan in the issue has sets
(a)then(b); consumed withfrom_substrait_planand executed onmainat039c0b99d5b:So the spec asks for an
i32holding0and1, and we return aUInt8holding1and2.Why this is easy to miss: the producer writes the same bitmask the consumer reads, so a DataFusion-to-DataFusion round trip agrees with itself and nothing looks wrong from inside DataFusion. It only diverges against an implementation that follows the spec.
(a)then(b)is the smallest case where the two differ — for(a, b)then(a)the bitmask and the index are both0, 1, which is why the existing round-trip tests pass either way.alamb commentedon Oct 25, 2026
But what are you trying to do?
namanjain24-sudo commentedon Oct 25, 2026
Honestly: nothing of mine broke. @alexandrefimov filed this from a Substrait conformance suite and I picked it up off the issue list.
The concrete gap is that the column we emit for a multi-set aggregate doesn't match what the spec says it holds, so a plan DataFusion produces is read differently by a consumer that follows the spec. I have not hit that in a real workload myself.
If that isn't worth the complexity, I am happy to close the PR and leave it to you and @alexandrefimov to decide whether the issue is worth doing at all.
alexandrefimov commentedon Oct 26, 2026
@alamb I'm building a Substrait conformance suite that runs the same plans through different implementations and compares their output schemas and rows. This issue came from testing multi-set aggregates: the trailing column has a different type and value in DataFusion than the spec defines.
DuckDB and Isthmus were mentioned as possible sources of SQL-generated plans for that comparison. I rechecked DataFusion main at
350ca765today, and the mismatch is still present.alamb commentedon Oct 26, 2026
I am sorry -- my feeble mind still doesn't understand this. It seems that the claim is that a datafusion generated plan consumed by substrait-java gives the wrong / different answer?
I didn't substrait-java could run queries
alexandrefimov commentedon Oct 26, 2026
No, substrait-java isn't executing anything here.
I'm feeding a standalone Substrait plan directly into DataFusion and comparing its output against the Substrait spec. DataFusion returns the wrong type and values for the grouping set column.
namanjain24-sudo commentedon Oct 26, 2026
No — substrait-java isn't part of it, and nothing outside DataFusion executes anything. Feed DataFusion a Substrait plan for
GROUPING SETS ((a), (b))throughfrom_substrait_planand the trailing column comes back as 1 and 2 where the spec says 0 and 1. That's the whole bug.Not worth your bandwidth though — I've said the same on the PR.
namanjain24-sudo commentedon Nov 4, 2026
untake