You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
bug: SparkCollectList/SparkCollectSet declare nullable list elements, diverging from Spark's containsNull = false #28
Spark declares both collect_list and collect_set as ArrayType(child.dataType, containsNull = false) — correctly, because both functions unconditionally drop null inputs (and the accumulators here hardcode ignore_nulls = true), so a null element can never appear in the result.
Since the elements are provably never null, the declared element field should be non-nullable, and the arrays the accumulators produce should carry the same non-nullable field.
There is a related inconsistency for nested inputs: return_type clones the input type verbatim (preserving any non-nullable inner fields, e.g. a non-nullable struct field), while the accumulators (ArrayAggAccumulator / DistinctArrayAggAccumulator + SingleRowListArrayBuilder) produce arrays whose nested fields are all nullable. So for nested types the declared and produced types disagree with each other, not just with Spark.
To Reproduce
In DataFusion Comet, which maps Spark's collect_set to SparkCollectSet, the produced List(Field { data_type: Int32, nullable: true }) does not match the Catalyst-derived schema List(Field { data_type: Int32 }), and a compensating cast has to be inserted per batch. Observed for every element type tested (Int8/16/32/64, Boolean, Utf8, Binary, Decimal128, Date32, Timestamp):
The nested-type disagreement between return_type and the accumulator output surfaces as Invalid argument error: column types must match schema types from AggregateExec output validation; Comet currently works around it by casting the input to an all-nullable variant before the aggregate (coerce_collect_child_nullability in its planner).
Expected behavior
return_type / return_field declares a non-nullable list element for both functions.
The accumulator output arrays carry a matching field, including for nested inputs, so no compensating cast is needed.
Additional context
Tracked on the Comet side in apache/datafusion-comet#4515 (a collection of functions whose Arrow return type drifts from Spark Catalyst's declared type). Sibling example already fixed the same way: apache/datafusion#22602 (width_bucket returning Int32 instead of Int64).
Hey,
I reproduced the main nullability mismatch on the latest main. For both collect_list and collect_set, DataFusion declares the list element as nullable even though the returned lists contain no null elements.
However, I could not reproduce the nested-type mismatch described in this issue. Using a struct with an explicitly non-nullable inner field, both the planned and runtime schemas preserved that field as non-nullable, and both queries completed successfully.
Could you please share the exact nested input shape or reproducer that causes the AggregateExec output validation error?
To exercise the nested mismatch, the regression declares Struct<required: Int32 non-null> but feeds a runtime StructArray whose required field metadata is nullable. Before the fix, accumulator state and output inherit the runtime nested metadata; the patch normalizes them to the declared type while keeping matching primitive buffers zero-copy @peterxcli Please let me know if you have any feedback. Thanks!
Activity
blinding-pixels commentedon Aug 28, 2026
Hey,
I reproduced the main nullability mismatch on the latest
main. For bothcollect_listandcollect_set, DataFusion declares the list element as nullable even though the returned lists contain no null elements.However, I could not reproduce the nested-type mismatch described in this issue. Using a struct with an explicitly non-nullable inner field, both the planned and runtime schemas preserved that field as non-nullable, and both queries completed successfully.
Could you please share the exact nested input shape or reproducer that causes the
AggregateExecoutput validation error?goutamadwant commentedon Aug 29, 2026
To exercise the nested mismatch, the regression declares
Struct<required: Int32 non-null>but feeds a runtimeStructArraywhoserequiredfield metadata is nullable. Before the fix, accumulator state and output inherit the runtime nested metadata; the patch normalizes them to the declared type while keeping matching primitive buffers zero-copy @peterxcli Please let me know if you have any feedback. Thanks!