Skip to content

perf: a shuffle over a native plan takes the JVM row shuffle for round-robin and nested hash partitioning #7073

Open

Description

@andygrove

What is the problem the feature request solves?

A shuffle whose child is a native Comet plan falls to the JVM columnar (row-based) shuffle when native shuffle rejects the partitioning. The batches then go Arrow to UnsafeRow to Arrow again, an avoidable round trip. CometShuffleExchangeExec.shuffleSupported picks native shuffle only if nativeShuffleFailureReasons is empty, and these partitionings are rejected by default:

  • round-robin (spark.comet.shuffle.native.partitioning.roundrobin.enabled, default false);
  • hash on struct, array or map keys (spark.comet.shuffle.native.partitioning.hash.nested.enabled, default false);
  • hash with an expression native cannot evaluate;
  • range partitioning on an unsupported type.

repartition(n) over a native plan is the common case, and it is round-robin.

Partly addressed since #5905 was filed: #6095 added an opt-in positional round robin (...roundrobin.positional.enabled) that skips hashing, but it only applies when Comet can prove the task replays its rows in the same order, so most plans still keep content-hash placement or the JVM path. #6607 converts Spark row input to Arrow for native shuffle, which is the other direction (Spark child, not native child).

Describe the potential solution

The native writer already accepts JVM Arrow batches through the synthetic ShuffleWriterInput scan. Add a CometPartitioning::PrecomputedIds(column) variant and append an Int32 partition-id column computed on the JVM side by a codegen projection, so Spark computes the id and its semantics are unchanged. Every partitioning native cannot evaluate then runs through native shuffle instead of the row path.

Separately, decide the default for nested hash keys. The stated blocker is that the nested murmur3 kernel's performance was unmeasured. native/spark-expr/benches/hash.rs now has struct, list and map shapes, so someone can run it against a flat-key baseline and either flip the default or point at the map path (hash_funcs/utils.rs, which slices one entry at a time) as the one to vectorize.

Additional context

Finding J1 in #5905. Related: #5397 and #5449 (round-robin on nested schemas), #6855 (nested-type conversion epic).

Activity

  1. JonSnow1807 commented on Nov 11, 2026

    @JonSnow1807

    I re-ran the nested hash key cases of CometShuffleBenchmark (--nested-hash-only) on main (e59d937), Spark 4.1.3, local[5], Ryzen 9 5900X, twice. JVM shuffle time divided by native shuffle time, the worse of the two runs, 5 / 201 partitions:

    Key Now #5788
    struct<int, string> 1.80 / 1.96 3.02 / 7.13
    array<int> 1.37 / 1.83 2.23 / 5.19
    struct<array<int>, string> 1.47 / 1.88 3.14 / 3.56
    array<struct<int, string>> 1.61 / 1.71 0.72 / 0.61
    struct<map<string, int>, int>, 1 entry 1.79 / 2.01 2.40 / 2.43
    same, 2-10 entries 0.96 to 1.05 1.37 to 1.79
    same, 2-50 entries 0.96 to 1.00 0.86 to 1.39

    Since #5778, native is faster for struct and array keys in every case, while maps with several entries are about even. (#5788 ran on other hardware, so only which side wins carries over.)

    Would you take a PR that turns native hash partitioning on by default for struct and array keys and keeps maps behind the flag? Or would you rather flip the whole flag, since maps are even rather than slower?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions