Skip to content

Support struct and array map keys in native MapSort #7632

Open

Description

@sam-1112

What is the problem the feature request solves?

On Spark 4.x, Comet's Rust MapSort implementation does not support maps with struct or array keys, such as MAP<STRUCT<a: INT, b: INT>, STRING> and MAP<ARRAY<INT>, STRING>.

#5590 and #5802 track a JVM codegen fallback that can keep the enclosing operator in Comet while Spark's generated code evaluates MapSort. This issue tracks a separate improvement: evaluating MapSort itself in Rust for these key types, without invoking the JVM dispatcher.

Describe the potential solution

Extend CometMapSort and the existing Rust map_sort kernel to support struct and array keys whose leaf types have Spark-compatible ordering. Preserve the existing scalar fast path and use the existing fallback behavior for unsupported key types. Leave floating-point leaves and non-default string collations out of the initial scope.

Add Spark parity tests that compare the ordered entries, not just map equality, and verify that MapSort runs in Rust with JVM codegen dispatch disabled. Cover nullable struct fields, nullable array elements, arrays with a common prefix and different lengths, and null or empty maps. For native shuffle, also verify that maps with the same entries in different orders get the same partition IDs as in Spark.

Additional context

Related to #5590 and #5802. The native implementation does not depend on #5802 landing first.

Willingness to contribute

None

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions