Repository navigation
Conversation
The GPU -> GPU converter stages through pinned host memory when a device pair has no working peer access. No test or benchmark reached that route: both skip unless peer access is granted. - The converter picks its route (peer or staged) once per conversion and passes it down, where each buffer copy looked it up before. - An internal registration overload takes options for the GPU -> GPU converter. `force_staged_route` stages every conversion, including one within a single device, so a one-GPU machine covers the route. - Tests: a mixed table staged within one GPU and across two GPUs in both directions, with buffers larger than one host block; host blocks all returned; the caller's device unchanged. The writer-ordering test now also runs over the staged route. - Benchmarks: the consumer fan-in over the staged route, and the same on up to four directed device pairs at once, with peak host staging bytes. No behaviour change for callers of register_builtin_converters(). Refs kevkrist#12. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… host memory memory::detail::staged_copy() moves a list of byte segments from one device to another through a ring of pinned host slots. The device-to-host leg runs on a stream of the source device and the host-to-device leg on a stream of the destination device, ordered only by events recorded before they are waited on, so the two legs of neighbouring chunks overlap and the calling thread waits once, for the last chunk. The ring holds at most `depth` chunks of `chunk_bytes` (4 x 4 MiB by default) and about the payload's size when the payload is smaller. It comes from the HOST pool of the destination's NUMA node, shrinks down to one block when the pool is short, and is pinned for the call when no pool is registered. After a failure the copy waits for the work it enqueued before returning the ring, and abandons the ring if that wait fails. memory::detail::read_device_integers() reads any number of 4- or 8-byte integers from a device with one wait, for callers that need sizes stored on the device before they can allocate. Nothing calls either function yet. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A GPU-to-GPU conversion between devices without a verified peer route copied
one column buffer at a time through host memory as large as the buffer, with
both legs on one stream and a host wait after every buffer and every STRING
column.
It now validates and measures the whole table, reads the sizes of all STRING
columns in one device round trip, allocates every destination buffer, and
moves them in one memory::detail::staged_copy() call. The column tree is
built on the host with the cudf::column constructor, so no cudf factory
launches a kernel on a stream the conversion does not control. The calling
thread waits at most three times per conversion: for the source device if the
writer is unknown, for the STRING sizes, and for the last chunk. Tables
without rows or columns are built on the host with no wait.
Host memory per conversion is bounded by the ring, which
register_builtin_converters(registry, staged_copy_settings{...}) configures.
On this route a column view must be whole. A view that starts past row 0, or
ends early over a column with children, is rejected with
std::invalid_argument in every build type instead of copying the wrong rows.
The same-device and verified-peer routes are unchanged.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… staged-copy tests Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ination GPU A staged copy takes its ring from the pool registered for the destination GPU's NUMA node. The tests registered theirs for node -1 and required it to be the only pool in the process, which another test's live HOST space would break. gpu_numa_node() is declared in the internal header so a test can register for the node the copy will look up. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
cudf 26.12 holds a column's null mask in cuda::device_buffer<bitmask_type>, which cannot be made from an rmm::device_buffer. The staged route now takes its masks from cudf::create_null_mask(), as the other converters do, so it builds against both mask types. Columns without rows get their empty column in the allocation pass, where a stream and a resource are at hand. The tests pass cudf a ::cuda::stream_ref, since rmm::cuda_stream has no view() in every supported RMM. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…the writer The staged route made a stream of the source device wait for the source's writer event while the destination device was current. The CUDA runtime recovers from that, so results were right, but the driver call fails first and compute-sanitizer reports "CUDA Stream does not belong to the expected context" once per cross-device conversion. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… past 2 GiB Converter: four concurrent conversions within one GPU, and two per direction between two GPUs; a HOST pool half the size of the default ring; a HOST pool with no block left, which must fail with rmm::out_of_memory, leave the pool's accounting unchanged and work again once memory is back. Copy helper: one segment of 2 GiB + 3 MiB + 5 B, checked at the offsets where chunk, ring and 32-bit boundaries fall. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements kevkrist#12 (design: kevkrist#18).
What
When two GPUs have no verified peer route,
convert_gpu_to_gpucopies the table through pinnedhost memory. That route copied one column buffer at a time, with both legs on one stream and a host
wait after every buffer and every STRING column.
It now:
GPU-to-host leg on a source-device stream and the host-to-GPU leg on a destination-device
stream, ordered by events;
cudf::columnconstructor.The calling thread waits at most three times per conversion, whatever the schema: for the source
device if the writer is unknown, for the STRING sizes, and for the last chunk. Tables without rows
or columns wait for nothing.
The copy itself is
memory::detail::staged_copy()in the core library (no cudf), with its owntests. The same-device and verified-peer routes are unchanged.
Behaviour changes on the staged route
STRUCT or DICTIONARY32 column, is rejected with
std::invalid_argumentin every build type. Arelease build copied the wrong rows before.
for smaller tables. It was the size of the largest column buffer.
register_builtin_converters(registry, staged_copy_settings{chunk_bytes, ring_depth}).Results
4 × A16, no peer access between any pair,
BM_ConvertGpuToGpuStaged*, median of 5, 8 tables of 32columns per iteration:
All 20 cells are faster, by 1.1× to 4.2×. The four GPUs of this host share one card; aggregate
throughput tops out near 8 GB/s for two and for four pairs, before and after. The targets in the
issue (≥ 85 GB/s across four cross-switch pairs, Sirius SF1000) need the reference machine and are
not measured here.
Sirius on the same host (
main83d6c0ee, TPC-H SF30, host-pinned tables, 12 samples per query):all 22 queries match the CPU results on 2 and 4 GPUs, and no query is slower. Time spent in
cross-GPU copies falls 14%. The suite is 2.2% faster on 2 GPUs and 0.8% faster on 4, the latter
within run-to-run noise.
compute-sanitizeron the converter's staged tests: the old route reportedCUDA Stream does not belong to the expected context270 times, because one stream served bothdevices; this route reports none.
Tests
test/memory/test_staged_copy.cpp: ring geometry and shrinking, block and slot boundaries, slotreuse with a delayed destination leg, failure after partial submission, abandoned ring, HOST
pool pressure and exhaustion, no HOST pool, cross-GPU in both directions, a segment over 2 GiB.
test/cudf/test_gpu_to_gpu_converter.cpp: fidelity of a 16-column mixed table within one GPUand across GPUs, writer ordering, wait counts, ring bound and settings, empty tables, rejected
slices, concurrent conversions, HOST pool pressure and exhaustion.
🤖 Generated with Claude Code