Repository navigation
Conversation
…em one at a time Hash, range, and hash round robin partitioning record one (batch, row) pair per row and build each output chunk with `interleave_record_batch`, which copies primitive and string columns a row at a time even when the rows sit together. A constant hash key, which an Iceberg write gets when Spark rebalances on a partition column holding one value per run, sends every row of a batch to one partition, so every chunk is one long run. `RowIterator` now splits each chunk into runs of consecutive rows of one buffered batch. When they average at least 128 rows it slices them out and concatenates them, and a chunk that is a single run reaches the writer as a slice. Scattered chunks and schemas with view types are gathered as before. The native shuffle guide no longer says positional placement is the only strategy that hands the IPC writer slices.
8a203b4 to
07765c2
Compare
andygrove
left a comment
There was a problem hiding this comment.
Thanks @comphead, this is a nice targeted win, and the per-chunk decision at flush is a clean answer to the open question in #7474.
I ran the benches on x86 (Ryzen 9 7950X3D) against the merge base, alternating the two binaries over two rounds. Your shuffle_hash_contiguous numbers reproduce closely: 0.46x and 0.59x for the constant key, and flat for the per-row key. I also added a case with the hash key clustered in runs of N rows, over your flat fixture, the 40-column nested struct fixture, and a schema with list<int64> (with null and empty lists), list<utf8>, map<utf8, int64> and struct<int64, list<utf8>>. From 136-row runs up, it's 0.45x to 0.73x on flat, down to 0.24x on nested and 0.27x on complex types. A task that is all one key runs at 0.02x on nested. That's great.
| average run (rows) | flat | nested structs | complex (list/map) |
|---|---|---|---|
| 32 | 1.00 / 1.01 | 0.99 / 1.01 | 1.00 / 1.00 |
| 96 | 1.02 / 1.02 | 1.00 / 1.00 | 1.00 / 1.00 |
| 120 | 0.91 / 1.02 | 0.87 / 0.99 | 0.83 / 1.00 |
| 136 | 0.72 / 0.73 | 0.64 / 0.56 | 0.51 / 0.56 |
| 192 | 0.66 / 0.65 | 0.60 / 0.45 | 0.44 / 0.44 |
| 512 | 0.58 / 0.51 | 0.45 / 0.24 | 0.35 / 0.27 |
| one key per task | 0.46 / 0.46 | 0.02 / 0.02 | 0.16 / 0.16 |
(PR time over base for place+gather of 65,536 rows, at 16 / 200 partitions.)
There's one band where it's slower. On the flat schema with runs of about 96 rows, place+gather goes from 908 µs to 930 µs. That's +2.4%, consistent across both rounds and at both 16 and 200 partitions. I think has_long_runs is scanning nearly the whole chunk there before falling back to the gather. Your sweep put flat break-even at about 32 rows and nested at about 64, so between 64 and 127 we pay for the scan and then take the slower path anyway. Would lowering MIN_AVG_RUN_ROWS to 64 work, since that's what you benchmarked with originally? It would also be good to add a clustered-key case near the threshold to shuffle_hash_contiguous, so the boundary stays covered.
Two other things:
-
Until now only
RowGroups, which is off by default, sent slices to the IPC writer. With this PR the default hash and range paths do too. From reading arrow-ipc 59.3.2, I believe it truncates every type Comet shuffles except the view types, which this PR keeps on the gather. Butcontiguous_rows_are_sliced_not_gatheredcovers onlyInt64,Utf8andBoolean, and compares in-memory batches. Could you add a case that sends struct, list, map and decimal columns through the slice path from sources with a non-zero offset, then round-trips them throughShuffleBlockWriterand the reader? An end-to-end check withcheckSparkAnswerinCometNativeShuffleSuite, usingrepartitionon a constant column with nested data columns, would also work. -
In
native_shuffle.mdthe new behavior is only described inside theRowGroupssection. Since it mainly affects hash and range partitioning, could you add a short note to the Hash Partitioning section, and point Range Partitioning at it?
Which issue does this PR close?
Closes #7474.
Rationale for this change
Hash, range, and hash round robin partitioning record one
(batch, row)pair per row and build each output chunk withinterleave_record_batch. Arrow copies primitive and string columns there one row at a time (interleave_primitive,interleave_bytes) and merges contiguous rows only ininterleave_fallback, which Boolean columns use. So a chunk whose rows are already one span of a buffered batch is still gathered row by row.That is the shape an Iceberg write gets when Spark rebalances on a partition column that holds one value per run, such as the
dsof an hourly or daily ETL. Every row hashes to one partition, and each buffered batch is a single run. In the production job described in #7474, partition interleaving was 54% of the native shuffle writer time in the map stage.DataFusion's
RepartitionExecgained the whole-batch case of this in 56.0.0. Itspartition_grouped_takepasses a batch through unchanged when all of its rows go to one partition (apache/datafusion#24598). This PR also covers runs that span buffered batches and output chunks.What changes are included in this PR?
RowIterator(partitioned_batch_iterator.rs) splits each output chunk into runs of consecutive rows of one buffered batch withchunk_by. When the runs average at least 128 rows, it slices them out of the buffered batches and concatenates the slices, and a chunk that is a single run reaches the writer as a zero-copy slice. Otherwise it gathers as before. Counting stops one run past the limit and before anything is sliced, so a scattered chunk costs only a short scan.partitioning_for_schemaalready applies to positional round robin, socontains_view_typebecomespub(crate).RunIteratorandRowIteratorshare the single-slice passthrough through a newconcat_slices.native_shuffle.mdno longer says positional placement is the only strategy that hands the IPC writer slices.shuffle_hash_contiguousbench group inbenches/shuffle_writer.rs, with a constant and a per-row hash key over 8192-row and 1000-row input batches.Range partitioning over clustered input takes the same path.
The threshold comes from a sweep with arrow 59.3.2 on Apple Silicon, over chunks of 8192 rows drawn from 16 batches. Slicing and concatenating beats
interleave_record_batchfrom an average run of about 32 rows on 4-column and 16-column flat schemas, and from about 64 rows on a nested schema (8 columns of a three-leaf struct inside two single-field structs, plus a list column). At 128 rows it is 2.5x to 3.3x faster on the flat schemas and 1.5x faster on the nested one, and for a whole batch it is 22x to 117x faster. I did not measure x86.How are these changes tested?
New unit tests in
partitioned_batch_iterator.rs:contiguous_rows_are_sliced_not_gathered: over sliced sources with anInt64, a nullableUtf8and a nullableBooleancolumn, chunks that are a whole batch, two runs, and part of a batch come out equal tointerleave_record_batch. The whole-batch chunk shares its source buffers, which shows the slice path ran.view_columns_are_gathered_even_when_contiguous: a contiguousUtf8Viewchunk long enough to slice is still gathered.Benchmarks ran on an 18-core Apple Silicon machine that was shared with other work, so the before and after binaries ran alternately and the numbers below are means over the rounds. They were measured with an earlier threshold of 64. Every case has runs of 1000 rows or more, or is scattered, so it takes the same path at 128.
shuffle_hash_contiguous, place and gather of 65,536 rows into 200 partitions, three rounds:The per-row cases only add the scan. Their differences have mixed signs within 3%, which is run-to-run noise.
shuffle_benchover a generated 4M-row Parquet file with 16 columns (8 strings, 7 longs, 1 int) and a constantds, 1600 partitions, zstd(1), two rounds:interleave timeds, beforeds, afterseq, beforeseq, afterEncode time does not change, so the IPC writer does not pay more for slices. zstd encoding takes most of the write time on this data, so the end-to-end gain here is smaller than the production share in #7474 suggests.
Input and commands
The input was generated with DuckDB: