Skip to content

[Incompatibility] Document array_join null handling differences #3178

Closed

Description

@andygrove

Summary

array_join is marked as Incompatible in Comet, but the specific incompatibility is not documented. This issue tracks documenting and potentially fixing the behavior difference.

Spark Specification

According to Spark's array_join behavior:

  • Concatenates array elements into a single string with delimiter
  • Without nullReplacement: null elements are ignored (skipped)
  • With nullReplacement: null elements are replaced with the specified string
  • Returns null if array or delimiter is null

Examples:

-- Without null replacement (nulls ignored)
SELECT array_join(array('hello', null, 'world'), ' ');
-- Spark returns: "hello world" (null is skipped)

-- With null replacement  
SELECT array_join(array('hello', null, 'world'), ' ', 'NULL');
-- Spark returns: "hello NULL world" (null is replaced)

Current Comet Implementation

Comet uses DataFusion's array_to_string function:

val arrayJoinScalarExpr =
  scalarFunctionExprToProto("array_to_string", arrayExprProto, delimiterExprProto)

Potential Incompatibilities

DataFusion's array_to_string may:

  1. Include null as empty strings in output (different from skipping)
  2. Have different null replacement behavior
  3. Handle delimiter differently

Verification Needed

Test these specific cases:

-- Null element without replacement
SELECT array_join(array('a', null, 'b'), ',');
-- Spark: "a,b"
-- Comet: ??? (may be "a,,b" or "a,b")

-- Null element with replacement
SELECT array_join(array('a', null, 'b'), ',', 'X');  
-- Spark: "a,X,b"
-- Comet: ???

Note: This issue was generated with AI assistance.

Activity

  1. vbhavh commented on Jan 15, 2026

    @vbhavh

    .take

  2. andygrove commented on Jan 25, 2026

    @andygrove
    MemberAuthor

    .take

    @vbhavh 👍

  3. 0lai0 commented on Apr 4, 2026

    @0lai0
    Contributor

    Hi, @vbhavh are you still doing this issue? If not, could I take this ?

  4. vbhavh commented on Apr 6, 2026

    @vbhavh

    hey @0lai0, just give me a week. I'll ping you if i won't be able to finish. :)

  5. 0lai0 commented on Apr 6, 2026

    @0lai0
    Contributor

    @vbhavh No worries! Happy to have you contributing. If anything's unclear, just let me know and we can figure it out together. Don't worry about the timeline, take as much time as you need.

  6. Visorgood commented on Aug 29, 2026

    @Visorgood
    Contributor

    @vbhavh @0lai0 hello! It seems a lot of time has passed. If any one of you still working on this, please let me know and I'll step back. Meanwhile, I will take it.

  7. Visorgood commented on Aug 30, 2026

    @Visorgood
    Contributor

    take

  8. Visorgood commented on Aug 30, 2026

    @Visorgood
    Contributor

    I looked into this and the outcome is a bit different from what the issue describes, so summarizing before opening a PR.

    The two cases in the issue already behave correctly. On the native path (allowIncompatible=true), array_join(array('a', null, 'b'), ',') returns a,b and array_join(array('a', null, 'b'), ',', 'X') returns a,X,b, both matching Spark. DataFusion's array_to_string documents its third argument as "If not provided, nulls will be omitted", which is the same rule Spark applies. The issue's "Current Comet Implementation" snippet quotes only the two-argument branch of CometArrayJoin.convert; the three-argument branch that forwards nullReplacement has been there since #1490.

    There is a real incompatibility, but it is a different one. Spark's ArrayJoin short-circuits to null on three conditions: a null array, a null delimiter, and a null nullReplacement. DataFusion's array_to_string only short-circuits on the first two – generate_string_array destructures just the array and the delimiter. A null null_string collapses to None, which it reads as "omit null elements", identical to passing no third argument.

    So the result diverges whenever nullReplacement evaluates to null, including when the array contains no nulls at all:

    arr nullrep Spark Comet native
    ["a", null, "c"] NULL NULL a,c
    ["a", "b", "c"] NULL NULL a,b,c
    ["a", null, "c"] X a,X,c a,X,c

    Repro on Spark 4.1.3, plan confirmed native via EXPLAIN (CometColumnarToRow → CometProject → CometNativeScan):

    CREATE TABLE aj(arr array<string>, delim string, nullrep string) USING parquet;
    INSERT INTO aj VALUES
      (array('a','b','c'), ',', NULL),
      (array('a',NULL,'c'), ',', NULL),
      (array('a',NULL,'c'), ',', 'X');
    SELECT array_join(arr, delim, nullrep) FROM aj;

    with --conf spark.comet.expression.ArrayJoin.allowIncompatible=true.

    The JVM codegen dispatcher path is unaffected and matches Spark throughout, so this only reaches users who have opted into the native path.

    Proposed fix, which I have working locally: wrap the three-argument array_to_string call in an IsNull(nullReplacement) guard inside CometArrayJoin.convert. It needs no native or protobuf change, and a non-nullable replacement (the common literal case) skips the guard entirely.

    Since the null-handling reason was the only non-collation entry for array_join in expression-audits/array_funcs.md, the fix also lets getSupportLevel return Compatible() for non-collated input, taking array_join off the opt-in path and making it native by default. Collated input stays Incompatible under #2190. Happy to split that part out if it is better to land the correctness fix on its own.

  9. Visorgood commented on Aug 30, 2026

    @Visorgood
    Contributor

    @andygrove hello! This is the first issue in the project I am willing to contribute to. If you have a chance, could you please have a brief look at my previous comment and suggest if the overall approach seems reasonable?

  10. 0lai0 commented on Aug 30, 2026

    @0lai0
    Contributor

    Hi @Visorgood, I’m not currently working on this, so please go ahead. Thanks for contributing!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions