Skip to content

bug: SparkCollectList/SparkCollectSet declare nullable list elements, diverging from Spark's containsNull = false #28

Open
apache/datafusion
#24767

Description

@peterxcli

Describe the bug

SparkCollectList and SparkCollectSet in datafusion-spark declare their result as a list with a nullable element field:

https://github.com/apache/datafusion/blob/main/datafusion/spark/src/function/aggregate/collect.rs

fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
    Ok(DataType::List(Arc::new(Field::new_list_field(
        arg_types[0].clone(),
        true,
    ))))
}

Spark declares both collect_list and collect_set as ArrayType(child.dataType, containsNull = false) — correctly, because both functions unconditionally drop null inputs (and the accumulators here hardcode ignore_nulls = true), so a null element can never appear in the result.

Since the elements are provably never null, the declared element field should be non-nullable, and the arrays the accumulators produce should carry the same non-nullable field.

There is a related inconsistency for nested inputs: return_type clones the input type verbatim (preserving any non-nullable inner fields, e.g. a non-nullable struct field), while the accumulators (ArrayAggAccumulator / DistinctArrayAggAccumulator + SingleRowListArrayBuilder) produce arrays whose nested fields are all nullable. So for nested types the declared and produced types disagree with each other, not just with Spark.

To Reproduce

In DataFusion Comet, which maps Spark's collect_set to SparkCollectSet, the produced List(Field { data_type: Int32, nullable: true }) does not match the Catalyst-derived schema List(Field { data_type: Int32 }), and a compensating cast has to be inserted per batch. Observed for every element type tested (Int8/16/32/64, Boolean, Utf8, Binary, Decimal128, Date32, Timestamp):

WARN shuffle/src/schema_align.rs: ShuffleWriter input schema mismatch on col[1]
'sort_array(collect_set(i), true)': child produced List(Field { data_type: Int32, nullable: true }),
catalyst declared List(Field { data_type: Int32 })

The nested-type disagreement between return_type and the accumulator output surfaces as Invalid argument error: column types must match schema types from AggregateExec output validation; Comet currently works around it by casting the input to an all-nullable variant before the aggregate (coerce_collect_child_nullability in its planner).

Expected behavior

  • return_type / return_field declares a non-nullable list element for both functions.
  • The accumulator output arrays carry a matching field, including for nested inputs, so no compensating cast is needed.

Additional context

Tracked on the Comet side in apache/datafusion-comet#4515 (a collection of functions whose Arrow return type drifts from Spark Catalyst's declared type). Sibling example already fixed the same way: apache/datafusion#22602 (width_bucket returning Int32 instead of Int64).

Activity

  1. blinding-pixels commented on Aug 28, 2026

    @blinding-pixels

    Hey,
    I reproduced the main nullability mismatch on the latest main. For both collect_list and collect_set, DataFusion declares the list element as nullable even though the returned lists contain no null elements.
    However, I could not reproduce the nested-type mismatch described in this issue. Using a struct with an explicitly non-nullable inner field, both the planned and runtime schemas preserved that field as non-nullable, and both queries completed successfully.
    Could you please share the exact nested input shape or reproducer that causes the AggregateExec output validation error?

  2. goutamadwant commented on Aug 29, 2026

    @goutamadwant
    Contributor

    To exercise the nested mismatch, the regression declares Struct<required: Int32 non-null> but feeds a runtime StructArray whose required field metadata is nullable. Before the fix, accumulator state and output inherit the runtime nested metadata; the patch normalizes them to the declared type while keeping matching primitive buffers zero-copy @peterxcli Please let me know if you have any feedback. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions