Skip to content

perf(cudf): pipeline host-staged GPU-to-GPU conversions - #290

Draft
Jedi18 wants to merge 8 commits into
NVIDIA:mainfrom
Jedi18:akhiljn/pipelined-staged-copy
Draft

Jedi18 wants to merge 8 commits into
NVIDIA:mainfrom
Jedi18:akhiljn/pipelined-staged-copy

Conversation

@Jedi18

@Jedi18 Jedi18 commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Implements kevkrist#12 (design: kevkrist#18).

What

When two GPUs have no verified peer route, convert_gpu_to_gpu copies the table through pinned
host memory. That route copied one column buffer at a time, with both legs on one stream and a host
wait after every buffer and every STRING column.

It now:

  1. validates the table and records every buffer, without touching the device;
  2. reads the character sizes of all STRING columns in one device round trip;
  3. allocates every destination buffer;
  4. moves all buffers as chunks through a ring of pinned host slots (4 × 4 MiB by default), the
    GPU-to-host leg on a source-device stream and the host-to-GPU leg on a destination-device
    stream, ordered by events;
  5. builds the column tree on the host with the cudf::column constructor.

The calling thread waits at most three times per conversion, whatever the schema: for the source
device if the writer is unknown, for the STRING sizes, and for the last chunk. Tables without rows
or columns wait for nothing.

The copy itself is memory::detail::staged_copy() in the core library (no cudf), with its own
tests. The same-device and verified-peer routes are unchanged.

Behaviour changes on the staged route

  • A column view must be whole. One that starts past row 0, or ends early over a STRING, LIST,
    STRUCT or DICTIONARY32 column, is rejected with std::invalid_argument in every build type. A
    release build copied the wrong rows before.
  • Host memory per conversion is the ring: at most 16 MiB by default, and about the table's size
    for smaller tables. It was the size of the largest column buffer.
  • New overload register_builtin_converters(registry, staged_copy_settings{chunk_bytes, ring_depth}).
  • A zero-row DICTIONARY32 column arrives with empty keys.

Results

4 × A16, no peer access between any pair, BM_ConvertGpuToGpuStaged*, median of 5, 8 tables of 32
columns per iteration:

Cell Before After Speedup
one pair, 2.5 MiB 23.2 ms 7.3 ms 3.2×
one pair, 40 MiB 34.6 ms 15.9 ms 2.2×
one pair, 648 MiB 242 ms (2.8 GB/s) 121 ms (5.6 GB/s) 2.0×
two pairs, 1.3 GiB 307 ms (4.4 GB/s) 169 ms (8.1 GB/s) 1.8×
four pairs, 2.6 GiB 358 ms (7.6 GB/s) 326 ms (8.3 GB/s) 1.1×

All 20 cells are faster, by 1.1× to 4.2×. The four GPUs of this host share one card; aggregate
throughput tops out near 8 GB/s for two and for four pairs, before and after. The targets in the
issue (≥ 85 GB/s across four cross-switch pairs, Sirius SF1000) need the reference machine and are
not measured here.

Sirius on the same host (main 83d6c0ee, TPC-H SF30, host-pinned tables, 12 samples per query):
all 22 queries match the CPU results on 2 and 4 GPUs, and no query is slower. Time spent in
cross-GPU copies falls 14%. The suite is 2.2% faster on 2 GPUs and 0.8% faster on 4, the latter
within run-to-run noise.

compute-sanitizer on the converter's staged tests: the old route reported
CUDA Stream does not belong to the expected context 270 times, because one stream served both
devices; this route reports none.

Tests

  • test/memory/test_staged_copy.cpp: ring geometry and shrinking, block and slot boundaries, slot
    reuse with a delayed destination leg, failure after partial submission, abandoned ring, HOST
    pool pressure and exhaustion, no HOST pool, cross-GPU in both directions, a segment over 2 GiB.
  • test/cudf/test_gpu_to_gpu_converter.cpp: fidelity of a 16-column mixed table within one GPU
    and across GPUs, writer ordering, wait counts, ring bound and settings, empty tables, rejected
    slices, concurrent conversions, HOST pool pressure and exhaustion.
  • Forced staging lets all but the cross-GPU cases run on one GPU.

🤖 Generated with Claude Code

Jedi18 and others added 8 commits November 15, 2026 19:18
The GPU -> GPU converter stages through pinned host memory when a device
pair has no working peer access. No test or benchmark reached that route:
both skip unless peer access is granted.

- The converter picks its route (peer or staged) once per conversion and
  passes it down, where each buffer copy looked it up before.
- An internal registration overload takes options for the GPU -> GPU
  converter. `force_staged_route` stages every conversion, including one
  within a single device, so a one-GPU machine covers the route.
- Tests: a mixed table staged within one GPU and across two GPUs in both
  directions, with buffers larger than one host block; host blocks all
  returned; the caller's device unchanged. The writer-ordering test now
  also runs over the staged route.
- Benchmarks: the consumer fan-in over the staged route, and the same on
  up to four directed device pairs at once, with peak host staging bytes.

No behaviour change for callers of register_builtin_converters().

Refs kevkrist#12.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… host memory

memory::detail::staged_copy() moves a list of byte segments from one device to
another through a ring of pinned host slots. The device-to-host leg runs on a
stream of the source device and the host-to-device leg on a stream of the
destination device, ordered only by events recorded before they are waited on,
so the two legs of neighbouring chunks overlap and the calling thread waits
once, for the last chunk.

The ring holds at most `depth` chunks of `chunk_bytes` (4 x 4 MiB by default)
and about the payload's size when the payload is smaller. It comes from the
HOST pool of the destination's NUMA node, shrinks down to one block when the
pool is short, and is pinned for the call when no pool is registered. After a
failure the copy waits for the work it enqueued before returning the ring, and
abandons the ring if that wait fails.

memory::detail::read_device_integers() reads any number of 4- or 8-byte
integers from a device with one wait, for callers that need sizes stored on
the device before they can allocate.

Nothing calls either function yet.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A GPU-to-GPU conversion between devices without a verified peer route copied
one column buffer at a time through host memory as large as the buffer, with
both legs on one stream and a host wait after every buffer and every STRING
column.

It now validates and measures the whole table, reads the sizes of all STRING
columns in one device round trip, allocates every destination buffer, and
moves them in one memory::detail::staged_copy() call. The column tree is
built on the host with the cudf::column constructor, so no cudf factory
launches a kernel on a stream the conversion does not control. The calling
thread waits at most three times per conversion: for the source device if the
writer is unknown, for the STRING sizes, and for the last chunk. Tables
without rows or columns are built on the host with no wait.

Host memory per conversion is bounded by the ring, which
register_builtin_converters(registry, staged_copy_settings{...}) configures.

On this route a column view must be whole. A view that starts past row 0, or
ends early over a column with children, is rejected with
std::invalid_argument in every build type instead of copying the wrong rows.

The same-device and verified-peer routes are unchanged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… staged-copy tests

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ination GPU

A staged copy takes its ring from the pool registered for the destination
GPU's NUMA node. The tests registered theirs for node -1 and required it to be
the only pool in the process, which another test's live HOST space would
break. gpu_numa_node() is declared in the internal header so a test can
register for the node the copy will look up.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
cudf 26.12 holds a column's null mask in cuda::device_buffer<bitmask_type>,
which cannot be made from an rmm::device_buffer. The staged route now takes
its masks from cudf::create_null_mask(), as the other converters do, so it
builds against both mask types. Columns without rows get their empty column
in the allocation pass, where a stream and a resource are at hand.

The tests pass cudf a ::cuda::stream_ref, since rmm::cuda_stream has no
view() in every supported RMM.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…the writer

The staged route made a stream of the source device wait for the source's
writer event while the destination device was current. The CUDA runtime
recovers from that, so results were right, but the driver call fails first and
compute-sanitizer reports "CUDA Stream does not belong to the expected
context" once per cross-device conversion.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… past 2 GiB

Converter: four concurrent conversions within one GPU, and two per direction
between two GPUs; a HOST pool half the size of the default ring; a HOST pool
with no block left, which must fail with rmm::out_of_memory, leave the pool's
accounting unchanged and work again once memory is back.

Copy helper: one segment of 2 GiB + 3 MiB + 5 B, checked at the offsets where
chunk, ring and 32-bit boundaries fall.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 16, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant