You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Support struct and array map keys in native MapSort #7632
On Spark 4.x, Comet's Rust MapSort implementation does not support maps with struct or array keys, such as MAP<STRUCT<a: INT, b: INT>, STRING> and MAP<ARRAY<INT>, STRING>.
#5590 and #5802 track a JVM codegen fallback that can keep the enclosing operator in Comet while Spark's generated code evaluates MapSort. This issue tracks a separate improvement: evaluating MapSort itself in Rust for these key types, without invoking the JVM dispatcher.
Describe the potential solution
Extend CometMapSort and the existing Rust map_sort kernel to support struct and array keys whose leaf types have Spark-compatible ordering. Preserve the existing scalar fast path and use the existing fallback behavior for unsupported key types. Leave floating-point leaves and non-default string collations out of the initial scope.
Add Spark parity tests that compare the ordered entries, not just map equality, and verify that MapSort runs in Rust with JVM codegen dispatch disabled. Cover nullable struct fields, nullable array elements, arrays with a common prefix and different lengths, and null or empty maps. For native shuffle, also verify that maps with the same entries in different orders get the same partition IDs as in Spark.
Additional context
Related to #5590 and #5802. The native implementation does not depend on #5802 landing first.
Activity