You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
array_join is marked as Incompatible in Comet, but the specific incompatibility is not documented. This issue tracks documenting and potentially fixing the behavior difference.
Spark Specification
According to Spark's array_join behavior:
Concatenates array elements into a single string with delimiter
Without nullReplacement: null elements are ignored (skipped)
With nullReplacement: null elements are replaced with the specified string
Returns null if array or delimiter is null
Examples:
-- Without null replacement (nulls ignored)SELECT array_join(array('hello', null, 'world'), '');
-- Spark returns: "hello world" (null is skipped)-- With null replacement SELECT array_join(array('hello', null, 'world'), '', 'NULL');
-- Spark returns: "hello NULL world" (null is replaced)
@vbhavh No worries! Happy to have you contributing. If anything's unclear, just let me know and we can figure it out together. Don't worry about the timeline, take as much time as you need.
@vbhavh@0lai0 hello! It seems a lot of time has passed. If any one of you still working on this, please let me know and I'll step back. Meanwhile, I will take it.
I looked into this and the outcome is a bit different from what the issue describes, so summarizing before opening a PR.
The two cases in the issue already behave correctly. On the native path (allowIncompatible=true), array_join(array('a', null, 'b'), ',') returns a,b and array_join(array('a', null, 'b'), ',', 'X') returns a,X,b, both matching Spark. DataFusion's array_to_string documents its third argument as "If not provided, nulls will be omitted", which is the same rule Spark applies. The issue's "Current Comet Implementation" snippet quotes only the two-argument branch of CometArrayJoin.convert; the three-argument branch that forwards nullReplacement has been there since #1490.
There is a real incompatibility, but it is a different one. Spark's ArrayJoin short-circuits to null on three conditions: a null array, a null delimiter, and a null nullReplacement. DataFusion's array_to_string only short-circuits on the first two – generate_string_array destructures just the array and the delimiter. A null null_string collapses to None, which it reads as "omit null elements", identical to passing no third argument.
So the result diverges whenever nullReplacement evaluates to null, including when the array contains no nulls at all:
arr
nullrep
Spark
Comet native
["a", null, "c"]
NULL
NULL
a,c
["a", "b", "c"]
NULL
NULL
a,b,c
["a", null, "c"]
X
a,X,c
a,X,c
Repro on Spark 4.1.3, plan confirmed native via EXPLAIN (CometColumnarToRow → CometProject → CometNativeScan):
CREATETABLEaj(arr array<string>, delim string, nullrep string) USING parquet;
INSERT INTO aj VALUES
(array('a','b','c'), ',', NULL),
(array('a',NULL,'c'), ',', NULL),
(array('a',NULL,'c'), ',', 'X');
SELECT array_join(arr, delim, nullrep) FROM aj;
with --conf spark.comet.expression.ArrayJoin.allowIncompatible=true.
The JVM codegen dispatcher path is unaffected and matches Spark throughout, so this only reaches users who have opted into the native path.
Proposed fix, which I have working locally: wrap the three-argument array_to_string call in an IsNull(nullReplacement) guard inside CometArrayJoin.convert. It needs no native or protobuf change, and a non-nullable replacement (the common literal case) skips the guard entirely.
Since the null-handling reason was the only non-collation entry for array_join in expression-audits/array_funcs.md, the fix also lets getSupportLevel return Compatible() for non-collated input, taking array_join off the opt-in path and making it native by default. Collated input stays Incompatible under #2190. Happy to split that part out if it is better to land the correctness fix on its own.
@andygrove hello! This is the first issue in the project I am willing to contribute to. If you have a chance, could you please have a brief look at my previous comment and suggest if the overall approach seems reasonable?
Activity
vbhavh commentedon Jan 15, 2026
.take
andygrove commentedon Jan 25, 2026
@vbhavh 👍
0lai0 commentedon Apr 4, 2026
Hi, @vbhavh are you still doing this issue? If not, could I take this ?
vbhavh commentedon Apr 6, 2026
hey @0lai0, just give me a week. I'll ping you if i won't be able to finish. :)
0lai0 commentedon Apr 6, 2026
@vbhavh No worries! Happy to have you contributing. If anything's unclear, just let me know and we can figure it out together. Don't worry about the timeline, take as much time as you need.
Visorgood commentedon Aug 29, 2026
@vbhavh @0lai0 hello! It seems a lot of time has passed. If any one of you still working on this, please let me know and I'll step back. Meanwhile, I will take it.
Visorgood commentedon Aug 30, 2026
take
Visorgood commentedon Aug 30, 2026
I looked into this and the outcome is a bit different from what the issue describes, so summarizing before opening a PR.
The two cases in the issue already behave correctly. On the native path (
allowIncompatible=true),array_join(array('a', null, 'b'), ',')returnsa,bandarray_join(array('a', null, 'b'), ',', 'X')returnsa,X,b, both matching Spark. DataFusion'sarray_to_stringdocuments its third argument as "If not provided, nulls will be omitted", which is the same rule Spark applies. The issue's "Current Comet Implementation" snippet quotes only the two-argument branch ofCometArrayJoin.convert; the three-argument branch that forwardsnullReplacementhas been there since #1490.There is a real incompatibility, but it is a different one. Spark's
ArrayJoinshort-circuits to null on three conditions: a null array, a null delimiter, and a nullnullReplacement. DataFusion'sarray_to_stringonly short-circuits on the first two –generate_string_arraydestructures just the array and the delimiter. A nullnull_stringcollapses toNone, which it reads as "omit null elements", identical to passing no third argument.So the result diverges whenever
nullReplacementevaluates to null, including when the array contains no nulls at all:arrnullrep["a", null, "c"]NULLNULLa,c["a", "b", "c"]NULLNULLa,b,c["a", null, "c"]Xa,X,ca,X,cRepro on Spark 4.1.3, plan confirmed native via
EXPLAIN(CometColumnarToRow→CometProject→CometNativeScan):with
--conf spark.comet.expression.ArrayJoin.allowIncompatible=true.The JVM codegen dispatcher path is unaffected and matches Spark throughout, so this only reaches users who have opted into the native path.
Proposed fix, which I have working locally: wrap the three-argument
array_to_stringcall in anIsNull(nullReplacement)guard insideCometArrayJoin.convert. It needs no native or protobuf change, and a non-nullable replacement (the common literal case) skips the guard entirely.Since the null-handling reason was the only non-collation entry for
array_joininexpression-audits/array_funcs.md, the fix also letsgetSupportLevelreturnCompatible()for non-collated input, takingarray_joinoff the opt-in path and making it native by default. Collated input staysIncompatibleunder #2190. Happy to split that part out if it is better to land the correctness fix on its own.Visorgood commentedon Aug 30, 2026
@andygrove hello! This is the first issue in the project I am willing to contribute to. If you have a chance, could you please have a brief look at my previous comment and suggest if the overall approach seems reasonable?
0lai0 commentedon Aug 30, 2026
Hi @Visorgood, I’m not currently working on this, so please go ahead. Thanks for contributing!