Skip to content

perf: slice contiguous rows in native shuffle instead of gathering them one at a time - #7636

Open
comphead wants to merge 1 commit into
apache:mainfrom
comphead:hash-shuffle-contiguous-runs
Open

comphead wants to merge 1 commit into
apache:mainfrom
comphead:hash-shuffle-contiguous-runs

Conversation

@comphead

@comphead comphead commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #7474.

Rationale for this change

Hash, range, and hash round robin partitioning record one (batch, row) pair per row and build each output chunk with interleave_record_batch. Arrow copies primitive and string columns there one row at a time (interleave_primitive, interleave_bytes) and merges contiguous rows only in interleave_fallback, which Boolean columns use. So a chunk whose rows are already one span of a buffered batch is still gathered row by row.

That is the shape an Iceberg write gets when Spark rebalances on a partition column that holds one value per run, such as the ds of an hourly or daily ETL. Every row hashes to one partition, and each buffered batch is a single run. In the production job described in #7474, partition interleaving was 54% of the native shuffle writer time in the map stage.

DataFusion's RepartitionExec gained the whole-batch case of this in 56.0.0. Its partition_grouped_take passes a batch through unchanged when all of its rows go to one partition (apache/datafusion#24598). This PR also covers runs that span buffered batches and output chunks.

What changes are included in this PR?

  • RowIterator (partitioned_batch_iterator.rs) splits each output chunk into runs of consecutive rows of one buffered batch with chunk_by. When the runs average at least 128 rows, it slices them out of the buffered batches and concatenates the slices, and a chunk that is a single run reaches the writer as a zero-copy slice. Otherwise it gathers as before. Counting stops one run past the limit and before anything is sliced, so a scattered chunk costs only a short scan.
  • The partition index is unchanged, so placement, memory accounting, and spilling are untouched. This answers the open question in Native hash shuffle gathers rows one at a time even when a whole batch goes to one partition #7474 with a choice made per output chunk at flush, rather than a mixed index of rows and runs.
  • A schema with a view type keeps the gather, because a sliced view array serializes every data buffer it shares. This is the rule partitioning_for_schema already applies to positional round robin, so contains_view_type becomes pub(crate).
  • RunIterator and RowIterator share the single-slice passthrough through a new concat_slices.
  • native_shuffle.md no longer says positional placement is the only strategy that hands the IPC writer slices.
  • A shuffle_hash_contiguous bench group in benches/shuffle_writer.rs, with a constant and a per-row hash key over 8192-row and 1000-row input batches.

Range partitioning over clustered input takes the same path.

The threshold comes from a sweep with arrow 59.3.2 on Apple Silicon, over chunks of 8192 rows drawn from 16 batches. Slicing and concatenating beats interleave_record_batch from an average run of about 32 rows on 4-column and 16-column flat schemas, and from about 64 rows on a nested schema (8 columns of a three-leaf struct inside two single-field structs, plus a list column). At 128 rows it is 2.5x to 3.3x faster on the flat schemas and 1.5x faster on the nested one, and for a whole batch it is 22x to 117x faster. I did not measure x86.

How are these changes tested?

New unit tests in partitioned_batch_iterator.rs:

  • contiguous_rows_are_sliced_not_gathered: over sliced sources with an Int64, a nullable Utf8 and a nullable Boolean column, chunks that are a whole batch, two runs, and part of a batch come out equal to interleave_record_batch. The whole-batch chunk shares its source buffers, which shows the slice path ran.
  • view_columns_are_gathered_even_when_contiguous: a contiguous Utf8View chunk long enough to slice is still gathered.

Benchmarks ran on an 18-core Apple Silicon machine that was shared with other work, so the before and after binaries ran alternately and the numbers below are means over the rounds. They were measured with an earlier threshold of 64. Every case has runs of 1000 rows or more, or is scattered, so it takes the same path at 128.

shuffle_hash_contiguous, place and gather of 65,536 rows into 200 partitions, three rounds:

Hash key Rows per batch Before After
constant 8192 693 µs 333 µs
constant 1000 702 µs 408 µs
per row 8192 981 µs 997 µs
per row 1000 1.369 ms 1.329 ms

The per-row cases only add the scan. Their differences have mixed signs within 3%, which is run-to-run noise.

shuffle_bench over a generated 4M-row Parquet file with 16 columns (8 strings, 7 longs, 1 int) and a constant ds, 1600 partitions, zstd(1), two rounds:

Hash key Write time interleave time Encode time Output
constant ds, before 1.301 s 0.151 s 0.94 s 335 MiB
constant ds, after 1.161 s 0.016 s 0.94 s 335 MiB
per-row seq, before 2.317 s 0.913 s 1.18 s 451 MiB
per-row seq, after 2.304 s 0.886 s 1.17 s 451 MiB

Encode time does not change, so the IPC writer does not pay more for slices. zstd encoding takes most of the write time on this data, so the end-to-end gain here is smaller than the production share in #7474 suggests.

Input and commands

The input was generated with DuckDB:

COPY (
  SELECT
    '2026-11-03' AS ds,
    md5(i::VARCHAR) AS record_id,
    sha256(i::VARCHAR) AS payload_hash,
    CASE WHEN i % 20 = 0 THEN NULL ELSE 'device-' || (i % 5000)::VARCHAR END AS device,
    ['US', 'DE', 'FR', 'JP', 'IN', 'BR', 'GB', 'CN'][(i % 8) + 1] AS country,
    (i % 256)::VARCHAR || '.' || ((i // 256) % 256)::VARCHAR || '.'
      || ((i // 65536) % 256)::VARCHAR || '.' || (i % 251)::VARCHAR AS ip,
    'app/' || (i % 37)::VARCHAR || '.' || (i % 11)::VARCHAR AS app_version,
    CASE WHEN i % 7 = 0 THEN NULL ELSE 'city_' || (i % 997)::VARCHAR END AS city,
    i::BIGINT AS seq,
    (1730000000000000 + i * 1000)::BIGINT AS event_time_us,
    (i * 2654435761 % 1000000007)::BIGINT AS user_id,
    (i % 1000)::BIGINT AS bucket,
    CASE WHEN i % 13 = 0 THEN NULL ELSE (i * 31)::BIGINT END AS bytes_in,
    (i * 17)::BIGINT AS bytes_out,
    (i % 86400)::BIGINT AS second_of_day,
    (i % 100)::INTEGER AS shard
  FROM range(4000000) t(i)
) TO 'constant_ds.parquet' (FORMAT parquet, ROW_GROUP_SIZE 1048576);
cd native
cargo bench -p datafusion-comet-shuffle --bench shuffle_writer -- shuffle_hash_contiguous
cargo build --release -p datafusion-comet-shuffle --features shuffle-bench --bin shuffle_bench
# --hash-columns 0 is the constant ds, 8 is the per-row seq
target/release/shuffle_bench --input constant_ds.parquet --partitions 1600 --partitioning hash \
  --hash-columns 0 --codec zstd --zstd-level 1 --warmup 1 --iterations 3

@comphead comphead added the run-benchmark-check Run the benchmark compile and lint check on this pull request instead of waiting for the merge queue label Nov 15, 2026
@github-actions github-actions Bot added enhancement New feature or request performance area:shuffle Shuffle (JVM and native) labels Nov 15, 2026
…em one at a time

Hash, range, and hash round robin partitioning record one (batch, row) pair
per row and build each output chunk with `interleave_record_batch`, which
copies primitive and string columns a row at a time even when the rows sit
together. A constant hash key, which an Iceberg write gets when Spark
rebalances on a partition column holding one value per run, sends every row
of a batch to one partition, so every chunk is one long run.

`RowIterator` now splits each chunk into runs of consecutive rows of one
buffered batch. When they average at least 128 rows it slices them out and
concatenates them, and a chunk that is a single run reaches the writer as a
slice. Scattered chunks and schemas with view types are gathered as before.
The native shuffle guide no longer says positional placement is the only
strategy that hands the IPC writer slices.
@comphead
comphead force-pushed the hash-shuffle-contiguous-runs branch from 8a203b4 to 07765c2 Compare November 15, 2026 19:51
@comphead comphead removed the run-benchmark-check Run the benchmark compile and lint check on this pull request instead of waiting for the merge queue label Nov 15, 2026

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @comphead, this is a nice targeted win, and the per-chunk decision at flush is a clean answer to the open question in #7474.

I ran the benches on x86 (Ryzen 9 7950X3D) against the merge base, alternating the two binaries over two rounds. Your shuffle_hash_contiguous numbers reproduce closely: 0.46x and 0.59x for the constant key, and flat for the per-row key. I also added a case with the hash key clustered in runs of N rows, over your flat fixture, the 40-column nested struct fixture, and a schema with list<int64> (with null and empty lists), list<utf8>, map<utf8, int64> and struct<int64, list<utf8>>. From 136-row runs up, it's 0.45x to 0.73x on flat, down to 0.24x on nested and 0.27x on complex types. A task that is all one key runs at 0.02x on nested. That's great.

average run (rows) flat nested structs complex (list/map)
32 1.00 / 1.01 0.99 / 1.01 1.00 / 1.00
96 1.02 / 1.02 1.00 / 1.00 1.00 / 1.00
120 0.91 / 1.02 0.87 / 0.99 0.83 / 1.00
136 0.72 / 0.73 0.64 / 0.56 0.51 / 0.56
192 0.66 / 0.65 0.60 / 0.45 0.44 / 0.44
512 0.58 / 0.51 0.45 / 0.24 0.35 / 0.27
one key per task 0.46 / 0.46 0.02 / 0.02 0.16 / 0.16

(PR time over base for place+gather of 65,536 rows, at 16 / 200 partitions.)

There's one band where it's slower. On the flat schema with runs of about 96 rows, place+gather goes from 908 µs to 930 µs. That's +2.4%, consistent across both rounds and at both 16 and 200 partitions. I think has_long_runs is scanning nearly the whole chunk there before falling back to the gather. Your sweep put flat break-even at about 32 rows and nested at about 64, so between 64 and 127 we pay for the scan and then take the slower path anyway. Would lowering MIN_AVG_RUN_ROWS to 64 work, since that's what you benchmarked with originally? It would also be good to add a clustered-key case near the threshold to shuffle_hash_contiguous, so the boundary stays covered.

Two other things:

  1. Until now only RowGroups, which is off by default, sent slices to the IPC writer. With this PR the default hash and range paths do too. From reading arrow-ipc 59.3.2, I believe it truncates every type Comet shuffles except the view types, which this PR keeps on the gather. But contiguous_rows_are_sliced_not_gathered covers only Int64, Utf8 and Boolean, and compares in-memory batches. Could you add a case that sends struct, list, map and decimal columns through the slice path from sources with a non-zero offset, then round-trips them through ShuffleBlockWriter and the reader? An end-to-end check with checkSparkAnswer in CometNativeShuffleSuite, using repartition on a constant column with nested data columns, would also work.

  2. In native_shuffle.md the new behavior is only described inside the RowGroups section. Since it mainly affects hash and range partitioning, could you add a short note to the Hash Partitioning section, and point Range Partitioning at it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:shuffle Shuffle (JVM and native) enhancement New feature or request performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native hash shuffle gathers rows one at a time even when a whole batch goes to one partition

2 participants