Repository navigation
feat(memory): REMOTE object-store memory tier (S3 spill via the native REST io_context) [stacked on #221] - #222
Draft
aminaramoon wants to merge 26 commits into
Draft
aminaramoon wants to merge 26 commits into
aminaramoon wants to merge 26 commits into
Conversation
Member
|
/ok to test 3b8b1d1 |
aminaramoon
force-pushed
the
io/s3b-remote-space
branch
2 times, most recently
from
October 29, 2026 05:38
13bd3e3 to
75417e7
Compare
Member
|
/ok to test 75417e7 |
aminaramoon
force-pushed
the
io/s3b-remote-space
branch
from
October 29, 2026 15:44
75417e7 to
65b46f7
Compare
Member
|
/ok to test 65b46f7 |
This was referenced Oct 29, 2026
aminaramoon
force-pushed
the
io/s3b-remote-space
branch
from
October 31, 2026 08:17
65b46f7 to
f3949dc
Compare
Member
|
/ok to test f3949dc |
aminaramoon
force-pushed
the
io/s3b-remote-space
branch
4 times, most recently
from
November 12, 2026 16:47
f941c00 to
55bd545
Compare
mike-wendt
pushed a commit
that referenced
this pull request
Nov 12, 2026
…LS; drop libcurl and kvikIO (IO split step 0b) (#227) > **Draft · stacked PR 2 of the IO split of #216 (step 0b).** Base: `main`. > **Review:** the single squashed commit on this branch. > > Stack (re-ordered): #209 (merged) → #218 (merged) → **this (0b: io_uring REST read + TLS)** → #219 (GDS read) → #220 (write API + DISK via io_context) → #221 (S3 writes) → #222 (REMOTE tier) → #224 (NUMA placement). ## Why this PR exists, and why it moved #221 used to carry the io_uring REST transport, the TLS layer, S3 writes and the kvikIO removal, late in the stack. A scan-replay benchmark of TPC-H SF1000 on S3 (`tpch_scan_bench`) showed the simpler S3 reactor beating that design: all 22 queries 59 s vs 80 s, q1 io-only 15.8 vs 9.8 GiB/s, up to 36 GiB/s with `min_mss` and 8 pinned reactors. So the READ half moves here, right after parity; S3 writes stay in #221. ## What **Transport (`src/io/rest/net`).** One io_uring per reactor, one keep-alive connection per slot; at most one primary + one cancel SQE per slot; slot never reused before both CQEs are reaped. Connects over a resolver thread's addresses (DNS never blocks the loop). Body bytes of a data GET land **directly into the destination iovecs** (pinned staging, host memory or `fs_cache` chunks); headers go through a scratch buffer. **TLS (`src/io/tls`, `include/cucascade/io/tls`).** `tls::config` (version bounds, `ktls = prefer | require | disable`, user-space RX via a built-in record layer or OpenSSL, io-wq placement, trust-store discovery, generation-safe session cache). After the OpenSSL handshake, installs kTLS RX+TX when policy and kernel allow (plain `SEND`/`RECVMSG`; tickets skipped; TLS 1.3 KeyUpdate re-keyed where the kernel can, else retire + retry); otherwise the built-in user-space record layer decrypts the ciphertext ring straight into the destination. `ktls = require` fails with `errc::ktls_unavailable`. **HTTP/1.1 codec (`src/io/rest/http`).** Zero-copy, allocation-free response parser, chunked body decoder, `Content-Range` value parser, per-field S3-focused dispatch table, iovec sink and a retry/reuse classifier. Request-head writer emits a single contiguous SEND and records field spans for the SigV4 signer. Chunked and close-delimited GET bodies are reported as `unsupported_framing` instead of being silently coalesced. **Reactor and pool (`src/io/rest`).** Adaptive physical GETs (`backlog / free connections` clamped to `[min_segment_bytes 4 MiB, max_segment_bytes 16 MiB]`), retry heap with backoff, jitter and Retry-After, bounded 403 re-signs, Content-Range validation, "server ignored Range", short-read retry, demand before prefetch. Device destinations stage through the HOST memory resource blocks and copy on the slice's stream with the slice's device current; the slot is held until the copy completes. Staging exhaustion **parks** (leases are counted per reactor; a read fails with `out_of_memory` only when the reactor holds no staging). **Control plane** (HEAD, footer probe, LIST, `resolve_footer_objects`) runs on the pool's own connections: least-backlogged reactor, at most half of a reactor's connections while data waits; never on the caller's thread (a private helper serves calls made before `start()` or from a reactor thread). **Wait modes.** `epoll_pwait2` over {eventfd, retry timerfd, ring fd} (default) or `io_uring_submit_and_wait_timeout` (`wait_mode = uring`, with `defer_taskrun` / `single_issuer`). **Generic exchange.** An optional request body (iovecs or generator) and `Expect: 100-continue` are already supported and tested against a loopback echo. This is the hook #221 (S3 writes) builds on; no S3 write semantics here. ### Pool behaviour: parity with the libcurl stack it replaces Checked against the curl source (`conncache.c`, `cf-socket.c`): a pooled connection is reused only if idle for less than `conn_max_age` (**20 s**, as `CURLOPT_MAXAGE_CONN`; #221 had silently moved this to 4 s) and it looks alive (zero-timeout poll: input pending or HUP with no response outstanding = dead). Dead or too-old connections are replaced **lazily, one slot at a time**, when a request needs them: there is never a mass reconnect. A request that dies on a reused connection before any response byte is retried once for free on a **fresh** connection. `warmup(bucket)` stays rate-limited to once per `conn_max_age` per bucket and pre-opens up to `max_connections` per reactor with real `LIST max-keys=0` exchanges on fresh connections. TCP keepalive 60/60 and `TCP_NODELAY` on. curl's upkeep is a no-op for HTTP/1.1, so nothing is sent while idle by default; `rest.upkeep_http1` (off) sends staggered zero-key LIST exchanges on idle connections every `upkeep_interval` instead. ### NIC-driven reactor sizing and pinning `rest_n_reactors` is a **total** (never per GPU). With `0`, the egress NIC (`rest.interface`, else the default route) is inspected: when its queue IRQs are steered to a strict subset of the NIC's NUMA node's cores (requested **and** effective masks narrow), the pool gets one reactor per IRQ CPU, each pinned to it before it creates its ring and raised to the highest priority the process may set (SCHED_FIFO if permitted, else nice; never widening the inherited affinity). Otherwise: the configured count (default 2), unpinned. The reactor's io-wq is kept off the pool's IRQ CPUs. Startup log reports the plan; `planned_reactor_cpus()` exposes it for test/introspection. ### `min_mtu` With jumbo frames, some pool connections negotiate small segments, which costs decryption time per byte. After the TCP connect, the effective MTU = `TCP_MAXSEG` + 40/60 (+12 with TCP timestamps) is compared with `rest.min_mtu`; too small → RST and reconnect on the next address, at most `mtu_reject_max_attempts` (8) times per request, then accept (`mtu_fallbacks`). Every close uses `SO_LINGER{on, 0}` (`close_with_rst`, default on) so no socket lingers in TIME_WAIT / FIN_WAIT. Stats: `mtu_rejects`, `mtu_fallbacks`, `mss_min`, `mss_max`. ### Removed - **kvikIO backend** (`refactor(io)!`): config, backend, tests and docs; the datasource registry has no catch-all any more (unclaimed paths return "no backend"). - **libcurl**: no longer a dependency of the REST path. ## Config Every #218 field keeps its name and default (`max_connections 64`, `conn_max_age 20 s`, `upkeep_interval 15 s`, `request_timeout_s 30`, stall detector 64 KiB/s × 30 s, `footer_probe_bytes 512 KiB`, list caps, retries). `ca_bundle_path` / `tls_verify` fill the new `tls` block. New: `tls`, `min/max_segment_bytes`, `receive_buffer_bytes`, `max_recv_iovecs`, `recv_waitall`, `queue_depth`, `sqpoll`, `single_issuer`, `defer_taskrun`, `register_ring_fd`, `iowq_max_unbound/bound`, `wait_mode`, `copy_poll_interval`, `hold_slot_during_copy`, `dns_ttl` / `accumulate` / `max_addresses`, `connect_timeout`, `keepalive_*`, `so_rcvbuf` / `sndbuf`, `busy_poll_us`, `tcp_user_timeout_ms`, `upkeep_http1`, `min_mtu`, `mtu_reject_max_attempts`, `close_with_rst`, `fan_out`, `interface`, `pin_to_irq_cpus`; `rest_n_reactors = 0` means auto. `config::validate()` rejects invalid values (including `max_retry_attempts` / `max_auth_retry_attempts` of 0, which used to be clamped silently).⚠️ API notes: `reactor_context` is no longer copyable (it holds a mutex; `make_shared` users are unaffected); `rest_ioctx::stats()` is new; libcurl is no longer a dependency. ## Behaviour differences from #221 that an embedder could notice A cut body is retried whole, not resumed; a 200 for a sub-range fails as "server ignored Range"; no link estimator/controller; the #221 fields `keep_warm_connections`, `enable_controller`, `link_gbps`, `ring_entries` (→ `queue_depth`), `tls_handshake_timeout` (→ `connect_timeout`), `write_split_bytes`, `expect_continue` do not exist here (the last two return with #221). ## Review round Two independent reviews (transport/TLS/pool; scheduler/destinations/contract/NIC) with mandatory reproduction: 0 blockers. Fixed with regression tests: staging exhaustion parked instead of failing (reproduced OOM on a 2 MiB pool); `upkeep_http1` busy-spin while a GET was in flight (1.9 M passes/s → < 100); control jobs no longer pinned to reactor 0 nor starving data (a footer batch delayed a 4 KiB read by 3 s → < 400 ms); `_inflight` leak on a throwing submit; control call from a reactor thread deadlocked; stale retry now goes to a fresh connection; CQEs reaped before deadline checks; kTLS KeyUpdate reply `EWOULDBLOCK` no longer fatal; idle drain loops while readable; pins re-intersected with the worker's affinity; pool plan kept per ioctx. Verified clean by the reviewers: coordinator / `queued_bytes` accounting, multi-GPU staging on 4 GPUs, the #218 public contract (base REST tests byte-identical and passing), a 400-exchange fault-injection run in all three ring modes. ## Tests `pixi run test` 10 / 10; 262 REST/TLS cases (8 skip: kTLS without `tls.ko`); 257 cudf cache tests through the REST backend; cudf-free build links without curl; 0 compiler warnings; lint clean. Loopback smoke (user-space TLS, 1 GiB in 16 MiB GETs, 2 reactors × 16 connections): 2.8–3.1 GiB/s. **Not tested here:** kTLS data path (no `tls.ko`; unit-tested with fakes), real S3, jumbo-frame MSS negotiation (the probe is tested through a seam), a NIC with steered IRQs (this host's are not steered; fake sysfs trees cover both cases). ## Branch state The 16 development commits have been **squashed into one** against the PR base. Verified at the squashed tip (Release, `sm_89`, `CUCASCADE_WARNINGS_AS_ERRORS=ON`, libcudf 26.12 nightly, 2× RTX 6000 Ada): `cucascade_tests` 166 cases, `cucascade_io_tests` 511 (1 skipped: `rest_ioctx over HTTPS with kTLS required fails without tls.ko`, as this host has no `tls.ko`), `cucascade_cudf_tests` 263, all passing; the benchmarks build. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Add the private cufile_api dlopen loader (libcufile.so.0, RTLD_NODELETE, read entry points only) and the process-wide cuFile platform probe (probe_platform / gds_available). The cuFile backend never links libcufile: it needs only cufile.h at configure time. find_path(cufile.h) gates the sources, the tests and the propagated CUCASCADE_HAS_CUFILE definition, independently of CUCASCADE_BUILD_CUDF, so a cudf-free build keeps the backend. The loader resolves read entry points only; write, buffer-registration and stream entry points are left out.
Add cufile_reactor / cufile_ioctx, a read-only GPUDirect Storage backend:
- device slices of at least gds_threshold bytes of a registered file are
read by synchronous cuFileRead calls of at most max_op_bytes (64 MiB),
split on 4 KiB file boundaries with the EOF tail read on its own; short,
zero and oversized returns are errors, a failing cuFileRead degrades to
POSIX. Destinations are never registered (cuFileBufRegister): every
read passes the exact device pointer.
- everything else goes through POSIX: host-only reads O_DIRECT (staged
through the scratch when unaligned), host+device pairs pread + H2D
copy, device-only reads double-buffered through two 8 MiB pinned scratch
slots on a reactor-owned non-blocking copy stream.
- a device slice whose stream still has queued work is parked behind an
event instead of blocking the worker; a parked slice of a request that
already failed settles at once instead of waiting for its stream, and
shutdown cancels parked slices and returns the pinned scratch to a
process-wide pool rather than calling cudaFreeHost (no device
synchronization).
- a read whose H2D copy was enqueued but whose event could not be recorded
waits for the copy stream before settling, so neither the destination
nor the scratch is released under a running copy.
- strict priority with bounded low service, per-pass byte quantum.
- opens reopen the buffered fd for O_DIRECT through /proc/self/fd, check
dev/inode before registering (retry once, then descriptor_mismatch_error)
and carry the buffered fd's local_stat identity and size.
- cufile_ioctx fans a vectored read with more than two slices out over
every reactor of its count-based pool.
Routing: io_backend::cufile and io_context_type::cufile;
io_config::{cufile_n_reactors, cufile} under CUCASCADE_HAS_CUFILE. With
backend=cufile, local regular files resolve to cuFile and s3:// stays on
REST; native routing is unchanged, and an explicit cufile
request in a build without cuFile is refused instead of served by another
backend. The reactor traits opt out of the prefetching cache. Vectored
device-read signatures are unchanged; their completion contract is now
documented at ioctx::device_readv_async_io and
datasource::device_read_ranges_async.
Cover read scheduling: another stream's read and a host read are served while a device slice waits for its held stream, shutdown cancels a parked slice promptly on the cuFile and the staged path (with pooled scratch already handed out), reads submitted before start are cancelled, and an active low request is parked for a high one with strict or bounded low service. Add the hidden read-only throughput probe [.cufile-bench] (CUCASCADE_IO_BENCH_FILE/BYTES/ROUNDS/REACTORS/BACKEND/ONLY/GDS_THRESHOLD). It validates that the operator-supplied input exists and is large enough and never creates, resizes or syncs it; the raw cuFile submission and write probes stay out.
Add topology_discovery::discover_storage_for_path and discover_storage_for_descriptor, returning a path_storage_info snapshot: the opened file's resolved path and device numbers, its mount (statx mount ID when the kernel reports it, else the longest component-boundary mount point of the file's device, with bind mounts and shadowed mounts handled and escaped mountinfo fields decoded), filesystem type / source / options, and for block filesystems the leaf disks reached through partition parents and device-mapper / md slaves (cycle-protected), each with drive class, PCI bus ID and NUMA node. Network and memory filesystems are classified without inventing devices; overlay / FUSE / pseudo filesystems, unreadable procfs or sysfs and unresolvable stacks are reported as unknown, which is distinct from missing, non-regular and inaccessible paths. Discovery is passive (procfs / sysfs only, no CUDA, no cuFile, no writes), never throws and caches nothing. It lives in its own source of the topology-only library; fixture procfs / sysfs roots are injected through a src-private header used by the tests only.
Add cufile::probe_path(path, config) and probe_object(object, config), and cufile_ioctx::path_capability(io_object) for objects it opened. They combine the effective compat / use_odirect policy, the process-wide platform probe, passive storage discovery of the opened descriptor (topology_discovery::discover_storage_for_descriptor, so the verdict describes the file actually read) and the open's O_DIRECT / registration outcome into a path_info with every piece of evidence and one verdict: unavailable, compatibility_only, direct_candidate or unknown, with a reason. Only the dstatusflags bit of the file's own storage class counts (NVMe, NVMe-oF, NFS, Lustre, WekaFS, GPFS, BeeGFS, VirtioFS), never any other bit; ext4 (not data=journal) and xfs are the supported local filesystems. A direct candidate is not a DMA guarantee. Unknown stacks and failed property queries stay unknown; compat OFF refusals are reported as unavailable with their cause. probe_path is explicitly active (it opens the cuFile driver and registers the file, reads nothing, writes nothing); verdicts are logged once per device and verdict. platform_info now distinguishes an opened driver whose properties could not be read (properties_available) from a compatibility-only one; gds_available() requires readable properties.
Document path_storage_status and the storage stack discovery that cufile::probe_path builds its per-file GDS verdict on.
Add the ioctx write API: write_mode / write_open_options / write_slice, open_io_object_for_write, synchronous host_write, asynchronous host / device / vectored / mixed writes, flush and flush_async, supports_write and supports_device_write, and the backend hooks behind them (defaults report std::errc::not_supported). Asynchronous forms are noexcept and report every error through the future. templated_ioctx gains the optional io_write_reactor_c contract with separate reactor_write_traits, so non-writing reactors keep compiling unchanged. Read and write dispatch share dispatch_slices; reads keep their batching, callback and priority behaviour. Write slices are validated (null, overlap, overflow, past off_t), never clamped to EOF, and cut at granule-aligned multiples of the reactor config's write_split_bytes. io_op_request::finish_success raises a writable object's size before the coordinator settles. A write invalidates its path in the writing ioctx's fs_cache and metadata store before dispatch and again before success or failure is visible (the completion is fully allocated before the write is dispatched, so nothing can fail between starting physical work and attaching the second pass); a truncating or replacing open invalidates too. fs_cache retires the path's generations and never hands an invalidated generation out again; a pending_invalidation holds an in-flight slot until it fires, so shutdown_cache waits for it. Retiring the generations is allocation-free and happens first, so an allocation failure can only defer the eager reclaim of their idle chunks, never leave a generation reusable. metadata_store gains a fixed 256-slot per-path-hash write epoch: objects are stamped at open, pre-write opens can neither observe nor publish, loads never join across a write, and entries are returned only to opens of the epoch they were published under. invalidate_written retires every profile of the path; targeted invalidate leaves the epoch alone. Writable objects carry an opened_handle identity.
Add io::not_implemented_error, a std::system_error with std::errc::not_supported whose message names the backend and operation and says "not yet implemented". The ioctx write front ends report it when the backend cannot write at all (a host-only writer asked for a device source still gets a plain not_supported), and so do the default write hooks. The REST context stays read-only and reports it from explicit write, flush and write-open hooks: synchronous hooks throw, asynchronous ones return a failed future, with no network request. The REST reactor gains static write-open / flush and member host_write stubs, and its enqueue fails write and flush slices instead of reading them as GETs. Because the stubs satisfy io_write_reactor_c structurally, reactor_write_traits<rest_reactor> is specialized to false/false next to the reactor, before any context instantiation. Tests cover every entry point through ioctx&, the direct hooks and direct reactor submission: exception type, code and message, no network side effects, and zero-size no-ops.
…e read start_device_copy enqueued the staged H2D copy and its completion event in one call. When copy_async failed part-way, or only the event record failed, the slot was settled at once: its staging block went back to the pool and the request resolved while the copy might still read the block and write the caller's destination. Record the event separately and, on any failure after the copy may have been queued, synchronize the caller's stream before settling, as the cuFile reactor already does. A process-wide test seam (uring::detail::set_copy_event_record_failure_for_testing) makes the record fail; the regression holds the caller's stream so the copy cannot have run, and checks the destination holds the file's bytes once the failed read settled.
A read future resolves with its coordinator's bytes_requested, the bytes of its slices as clamped at dispatch to the size recorded on the io_object. That count is the bytes delivered only because every reactor path fails a transfer that comes up short of the clamped range instead of settling it as a success: uring resubmits short reads and fails a zero-byte one with EIO (also when its O_DIRECT range was widened), and the cuFile reactor fails a short cuFileRead, staged pread or host pread with EIO. Nothing pinned that contract. Document it on grouped_coordinator and device_readv_async_io, and test it for uring (O_DIRECT and buffered) and both cuFile read paths: reads running past EOF resolve with exactly the bytes up to EOF (scalar and vector forms, bytes checked), a read starting at EOF resolves 0, and reads of a file truncated after it was opened fail with std::errc::io_error for device and asynchronous host reads, while a part the file still holds reads in full and the synchronous host read reports what pread delivered.
cudaStreamPerThread names the per-thread default stream of whichever thread uses the handle. The reactors interpret a device slice's stream on their worker thread: the uring reactor enqueues the H2D copy of a device read (and now the D2H copy of a staged write) on it, and the cuFile reactor records its stream gate on it. With cudaStreamPerThread both ordered the transfer after the worker's own, empty, per-thread stream, so a read could land before work the caller had queued ahead of it (and be overwritten by it), and a write could read its source before the caller's producer ran. templated_ioctx's dispatch now replaces every cudaStreamPerThread of a request's device slices, on the submitting thread, by a non-blocking stand-in stream (one per device and request) that waits there on an event recorded on the caller's per-thread stream; the slice keeps the stand-in alive. No per-thread handle crosses threads any more. The fs_cache's cache-hit device copies, enqueued on the caller's stream by the submitting thread, were retired through a completion lane keyed by stream handle; cudaStreamPerThread has one handle for every thread's stream, so one thread's completion could retire another thread's copies (releasing pins and resolving the read before its DMA finished). A cache-hit read on the per-thread stream is now retired by its own event, recorded on the submitting thread after its copies. The legacy default stream needs nothing: it is one stream per device and synchronizes with every blocking stream. Regression tests hold the caller's per-thread (and, for coverage, legacy) default stream on the host, queue a memset of the destination behind it and check the device read's bytes, for the uring reactor and for both cuFile read paths; a two-thread test holds one thread's per-thread stream under a cache-hit read and checks another thread's cache-hit read neither retires nor waits for it.
The io_uring reactor models io_write_reactor_c. A write slice is split at the write granule (max(4 KiB, page size)) into an unaligned head, an aligned body and an unaligned tail: the body goes O_DIRECT straight from a congruent host source or through the pinned staging blocks (a D2H copy on the caller's stream for device sources, a memcpy for host ones), head and tail through the page cache. Writes use write / writev / write_fixed SQEs; short writes are resubmitted, a zero-byte completion is EIO, and O_DIRECT or fixed-buffer runtime errors fall back to buffered I/O as reads do. Flush is an IORING_OP_FSYNC with IORING_FSYNC_DATASYNC; the synchronous host_write and flush run on the caller. write_split_bytes (default 32 MiB) spreads large slices over the selected reactors. The worker never blocks on a caller's stream: pending D2H and H2D copies are polled with a capped CQE wait, and staged-write bytes whose copy is pending count in queued_bytes(). start() waits for the worker's ring setup and rethrows its error (e.g. EPERM under seccomp); requests enqueued before or after such a failure fail with that error, and start() may be retried. Clean shutdown still cancels with operation_canceled. Writable local objects grow atomically as writes complete and carry an opened_handle identity. The write open anchors its O_DIRECT descriptor to the already-created buffered one through /proc/self/fd instead of reopening the path, never repeats the destructive create / truncate, and fails with descriptor_mismatch_error when a pathname fallback cannot prove both name the same file; a test seam renames the path at that point. Tests cover open modes, aligned / unaligned host and device scalar, vector and mixed writes, holes and size growth, stream ordering, priorities, concurrent block-sharing writers, shutdown and pre-start cancellation, the mincore direct-write check, stalled streams, short writes, RLIMIT_FSIZE, sibling errors, seccomp start-up failure, the anchored write open, and fs_cache / metadata invalidation by writes.
The cuFile reactor models io_write_reactor_c. Write opens create / truncate the buffered fd per write_mode, preallocate best-effort, and reopen the O_DIRECT fd from it through /proc/self/fd (never by path; a fallback that opens another file fails with descriptor_mismatch_error rather than repeat the destructive open), then register it with cuFile under the usual compat rules. Writable objects grow atomically and carry an opened_handle identity. Writes split at the write granule. On a real GDS platform the aligned body of a device source of at least gds_threshold goes to chunked synchronous cuFileWrite after the slice passed its stream gate; head, tail, smaller bodies and, on a compatibility-only platform, every device body are staged through the double-buffered pinned scratch by D2H copies on the reactor's copy stream and written with pwrite (O_DIRECT for aligned chunks). Large host bodies congruent with their offset go O_DIRECT from the caller; other host sources go through the page cache with writeback started at once, and a host-only vectored write goes to one reactor. Short writes resubmit, zero-byte writes are EIO, flush is fdatasync. Staging copies drain before the scratch is reused or a request settles, including when an event record fails after a copy was queued. cufile_api resolves cuFileWrite, cuFileBufRegister and cuFileBufDeregister. config::register_min_bytes (default 0) opts into per-transfer buffer registration through a process-wide, reference-counted table keyed by base address. Every non-empty cuFile read and write takes part in that table, also with the knob at 0 (which only stops a context from creating registrations), so a context that registers never covers memory another context's unregistered transfer is using. write_split_bytes defaults to 0 for cuFile. Tests cover every GDS / staged / compat / unregistered / buffered write path, writes ordered after the caller's per-thread and legacy default streams, default and opt-in contexts sharing a buffer, registration sharing and exclusion, caller stream ordering, progress of unrelated host writes, shutdown cancellation, short and error paths and 8-byte-aligned host-block vectors. The practices doc gains a writes section (measured in compatibility mode only).
Pin null device streams to cudaStreamLegacy before dispatch, and gate cuFile on a recorded legacy-stream event instead of assuming cudaStreamQuery captures blocking-stream dependencies. Canonicalize cache-hit retirement lanes too. Cover dispatch sentinels, null read/write streams and PTDS-built callers on uring and direct/staged cuFile paths. The deterministic enqueue seam fails without the fix; the normal legacy-built backend already passes the PTDS-caller test. An isolated PTDS-built cuFile backend reproduces early completion and stale writes without the fix (0ms, 0x11 written), and waits ~700ms with the fix (0xab written).
DISK memory spaces now read and write their files through a per-space
std::shared_ptr<io::ioctx> instead of the bespoke idisk_io_backend and its
pipeline worker.
- disk_memory_space_config gains a defaulted io_context, and
reservation_manager_configurator::set_disk_mounting_point a defaulted
fourth argument. memory_space gets DISK-only, mutex-guarded
get_io_context(), set_io_context() (non-null) and
get_or_init_io_context(make_default); a failed or null factory installs
nothing, and in-flight conversions keep their own reference. The core only
forward-declares ioctx: no core -> io link.
- cucascade/cudf/disk_io_context.hpp: make_disk_io_context() builds and
starts an automatic / uring / cufile context (uring owns private portable
pinned staging, 64 MiB per reactor), and default_disk_io_context() is the
weakly cached, process-shared default adopted lazily on first conversion.
automatic = cuFile with real GDS, else uring when this process may set up
io_uring (also falling back when the reactors' own ring setup fails at
start), else the cuFile reactor's POSIX path; forced uring fails with the
setup errno. The cuFile paths are gated on CUCASCADE_HAS_CUFILE.
- The four HOST/GPU <-> DISK converters open one object and issue one
vectored request per table; the GPU read allocates the whole buffer tree
before submission, waits once, and records the writer event. Every
allocation outlives its I/O on every unwind path; spill files keep their
RAII removal and layout; no durability flush in the spill path. The
redundant post-read host stream sync is gone: a completed device read is
host-observed.
- A cudf build now requires CUCASCADE_BUILD_IO (configure-time diagnostic);
cudf-free and topology-only builds are unaffected.
- Removed: include/cucascade/data/{disk_io_backend,io_backend_registry}.hpp,
src/data/{pipeline_io_backend.cpp,io_worker.hpp,io_backend_registry.cpp,
io_backend_internal.hpp} and their tests.
AGENTS.md describes the io_context-based DISK tier.
BREAKING CHANGE: the custom-backend memory_space constructor and getter,
idisk_io_backend, io_batch_entry and the io backend registry are removed.
Three-field disk_memory_space_config aggregates and three-argument
set_disk_mounting_point calls still compile; pass an io::ioctx (or nothing,
for the shared default) instead of a backend.
Keep the source device current from wait_for_writer through the write: a null stream otherwise named two different default streams and the spill could read stale source bytes before its writer finished. Reject an explicit stream of another device with cucascade::logic_error before generating a file path.
Both regression cases ran on the 4xA100 host for uring and cuFile compatibility mode. Without this fix the null-stream test spills stale bytes on both backends; foreign streams yield a CUDA error on uring or no error on cuFile. With it, both cases pass (22 assertions).
- test_disk_randomized: randomized mixed / nested / empty tables with non-repeating data, round-tripped through HOST and GPU on the default context and on forced uring and cuFile contexts; only forced uring is skipped, and only where io_uring is proven unusable in this process. - benchmark_disk_converter measures the DISK tier per io context with backend-neutral names (automatic, forced cufile, optional forced uring), CUCASCADE_BENCH_DISK_DIR, cold reads and an explicit benchmark-only fdatasync for comparable write numbers. No throughput assertions.
…eries
The S3 write path needs PUT (object, UploadPart), POST
(CreateMultipartUpload, CompleteMultipartUpload) and DELETE
(AbortMultipartUpload) requests, most with a sub-resource query
(?uploads, ?partNumber=N&uploadId=U, ?uploadId=U).
- request_method gains PUT, POST and DELETE, plus to_string();
- request_spec describes one object-level request (method, pre-encoded
canonical sub-resource query, whether it has a body);
- request_authorizer::authorize_request(spec, timeout): a new virtual whose
default delegates plain GET / HEAD to authorize() (existing authorizers
keep working) and throws credential_error for anything else;
- the SigV4 header and presigned authorizers implement it: the query is
merged into the canonical request, a body is signed as UNSIGNED-PAYLOAD
(sha256("") without one), and X-Amz-* keys in the query are rejected
(no signing-parameter smuggling).
Tests ([s3][sigv4]): canonical requests and signatures for every write
request kind against independently computed SigV4, query validation, and
the mock authorizer's recording of request specs.
S3 writes without a transport (src/io/rest/s3/multipart): nothing here performs I/O, allocates pinned memory or touches CUDA. - protocol helpers: the S3 limits (10000 parts, 5 MiB minimum part, 5 GiB part / single PUT), part numbering for a fixed part size, the canonical sub-resource queries of CreateMultipartUpload / UploadPart / CompleteMultipartUpload / AbortMultipartUpload, the Complete request XML, and parsers for InitiateMultipartUploadResult, CompleteMultipartUploadResult (including a 200 OK carrying an <Error> body, which S3 says to retry) and part ETags; - upload_state: the thread-safe state machine of one object being written (parts assembled from out-of-order writes, uploaded, retried, committed as one PUT for an object of at most one part or as a multipart upload otherwise, or aborted), which the REST reactor drives. This includes NVIDIA#221's fix 5ff07f6: a Complete 2xx without a result document fails. interpret_complete_response used to accept any 2xx body that was not an S3 <Error> as success, so a proxy page or keep-alive whitespace yielded an empty ETag and the object counted as committed although the commit was never acknowledged. A CompleteMultipartUploadResult element is now required (attributes such as xmlns accepted); anything else is http_bad_message and not retried. A result without an ETag is still accepted, for S3-compatible stores that omit it. Tests ([s3]): part planning, request and response XML ("multipart: CompleteMultipartUpload response (200 OK + <Error>)"), ETag handling, and upload_state transitions under concurrent writers, failures and aborts.
…s in the transport What the S3 write path needs from the io_uring transport before it sends PUT and POST bodies: - The stall detector counts request body bytes sent as well as response bytes received. It used to count received bytes only, so an upload slower than one stall window was cancelled while it was still sending (curl's low-speed check measures the transfer in either direction). A server that never answers still stalls. - exchange_result::request_sent: the head and the whole body were written. With a transport error and no response byte the server may have acted on a request whose answer was lost; CreateMultipartUpload must never be re-sent then, and a conditional commit that retries into a 412 cannot tell whether its own earlier attempt took effect. - exchange_result::error_body_complete and response_meta::amz_error_code: a drained error body's S3 <Code> (http::extract_error_code), and whether the body was drained to its end. A 404 alone does not say NoSuchUpload from NoSuchKey, and a cut body must not be read as a known code. Accepted (2xx) bodies stay the caller's to interpret. - exchange::continue_timeout: with Expect: 100-continue, the held body goes out anyway when no interim or final response arrived in time (a server that ignores Expect; curl's EXPECT_100_TIMEOUT). The wait is a POLL_ADD that the deadline cancels softly (both CQEs reaped before the body is sent), so a user-space TLS stream's receive state is never disturbed. - The user-space TLS stream seals stream_options::tx_batch_records (16) records per ciphertext SEND instead of a fixed 3: one SEND is in flight at a time (record order), so a 16 MiB part took ~340 SEND round trips; now ~64. The FIFO holds 17 records; it is not value-initialised, so a connection that never sends a body touches one page of it. Tests ([rest][net]): a slowly read 8 MiB upload under a 1 s stall window completes (and fails without the send-side progress), a silent server still stalls, request_sent on a refused connection and on a dropped response, the error code of a drained 404 and an incomplete cut body, a held body sent after continue_timeout with the connection reused after, and the sealer's record budget per stream_options.
The write half of NVIDIA#221's loopback S3 fixture, re-applied onto the read server of the io_uring reactor (HTTPS mode, mid-body KeyUpdate and drop policies, chunked GET bodies, trailing garbage, data-GET waits are kept): - an in-memory object store: PutObject (If-None-Match: * -> 412 when the key exists), CreateMultipartUpload, UploadPart, CompleteMultipartUpload (part order, ETags and the minimum part size validated), AbortMultipartUpload and DeleteObject; written keys are what later GET / HEAD / LIST see, missing_key_404 turns other keys into NoSuchKey; - write faults (write_kind, write_fault: status, ok_with_error, ok_with_body, drop_mid_body, drop_after_body, hold), Expect: 100-continue handling (honor / ignore / reject), keep-alive whitespace before the Create / Complete XML; - counters (put_object_count, create_upload_count, upload_part_count, complete_upload_count, abort_upload_count, delete_object_count, active_upload_count, ...), request records and put_stored_object seeding. ETags are S3's: the MD5 of a body for a single PUT and a part (hashed outside the store lock, so uploads of several connections hash in parallel), md5(concat(binary part MD5s))-N for a completed upload, so a client can verify a commit against its part ETags. Every public name is NVIDIA#221's (the REMOTE tier's test utilities use them), s3_error_body is public, and a constructor overload keeps the read-only HTTPS call sites (object, faults, listing, scripts, tls) unchanged. No library change. Tests ([s3]): test_loopback_s3_server.cpp, the fixture itself, against independently signed requests and raw sockets.
…art upload, commit on flush Enable writes on the REST backend, replacing the not-implemented stubs: rest_reactor models io_write_reactor_c with supports_device_write (the reactor_write_traits specialization is gone), so rest_ioctx reports supports_write() and supports_device_write(). The upload state machine (s3::upload_state) decides; the reactor's worker moves bytes and settles every claim exactly once. The decision logic is NVIDIA#221's one-for-one; the plumbing is rebuilt on the slot / exchange model of the read reactor. - Open: create_or_truncate and create_new (If-None-Match: * on the final PUT / Complete); create_or_open and open_existing are not_supported. No request at open; size_hint raises the part size past 10000 parts. - Writes land in parts of write_split_bytes. A host slice covering a part is uploaded from the caller's buffer; anything else is copied (memcpy on the worker, or D2H on the caller's stream, polled like the H2D copies of reads) into a part buffer from the context's host resource; the write completing a part uploads it. rest_ioctx cuts slices at part boundaries. - Part buffers count as staging: on exhaustion a write parks while write work or staged reads of the reactor are in flight and retries when they settle or a buffer is released (on any thread: the release wakes the loop); with nothing in flight it fails with rmm::out_of_memory. A parked staged read no longer ends the submit pass (it used to `break`): the write calls behind it, which return the blocks, still get their connections, and wait() never blocks while a write call could start; otherwise a burst of parts and a staged read sharing a small pool deadlocked. Write and flush requests have their own intake, expanded while reads are parked (most writes need no staging). - flush / flush_async commit (one flush slice on the least-backlogged reactor; rest_ioctx::flush_io waits on flush_async_io): the remaining parts (gaps zeroed; never-written parts from a static 1 MiB zero block in windows of 64 iovecs, so a zero body can no longer be freed while it is sent: NVIDIA#221's b9d9090), then CompleteMultipartUpload, or one PUT for an object of at most one part. The path's cached bytes and metadata are invalidated once the commit settled (ioctx::invalidate_cached is protected for that). - Each S3 request is an s3_call with its own retry envelope on the worker's connections. A call is owned by the worker's queues or slot and owns its job (a part, a flush, a Create), never the reverse (NVIDIA#221's a16aa5a): the last call of a failed commit releases the job, and with it the session whose destruction aborts the upload. Work queued during a pass, including a Create's follow-up UploadParts and the waiters its settlement hands back, starts in that pass (NVIDIA#221's fc28071 invariant). - Rules (owner's decisions): PUT / UploadPart re-sign a 403 up to max_auth_retry_attempts; Create / Complete / Abort stay final on 403. CreateMultipartUpload is never re-sent once it went out whole and its answer was lost, except for the free stale retry (a pooled connection that died before any response byte: the request was not read). The object's session remembers, across flushes, that a commit request lost its answer: a create_new commit answering 412 then reports io_error ("commit outcome unknown"), never EEXIST, and a Complete answered NoSuchUpload is checked with one HEAD against md5(concat(binary part MD5s))-N of the recorded part ETags; a mismatch (or non-MD5 part ETags) fails the flush as "outcome unknown". Without a lost answer a NoSuchUpload fails the flush. A 409 ConditionalRequestConflict on a conditional commit is retried. A 2xx Complete without a result document fails (NVIDIA#221's 5ff07f6). Never a false commit success. - Cross-reactor work: parts of one object may run on several reactors; work parked on the object and released by a settlement runs on the settling reactor in that pass when it parked there, and is handed back to the reactor that parked it otherwise (an intake queue and an eventfd wake; it starts in the pass that wakes it), so the parts of a burst dispatched before the UploadId exists upload on every reactor instead of all on the one that created it. The abort of an object destroyed uncommitted goes through an abort sink owned by rest_ioctx that posts a control job (signed from a request spec, so DeleteObject needs only a new kind) to a running reactor. Lifetime: the sink's close (first in ~rest_ioctx) makes every write object still alive give back its part buffers while the host memory resource lives; such an object then fails its writes and flushes with operation_canceled. - Config: write_split_bytes (16 MiB), expect_continue (off), continue_timeout (1 s), complete_timeout_s (300), max_write_connections (0 = all). Stats: puts, put_bytes, upload_parts, upload_part_bytes, zero_part_bytes, creates, completes, complete_error_retries, aborts, commits, commit_failures, write_failures, d2h_copies, part_buffer_waits, part_buffers_live. Test hooks: on_call_started / on_call_finished. - The reactor_worker class moves to src/io/rest/reactor_worker.hpp (no behaviour change) so the write path is its own TU. Tests ([rest][write]): every NVIDIA#221 case, including the regressions "aborts an upload whose commit failed once the object is destroyed" (a16aa5a), "a 200 Complete without a result document fails the flush; flushing again commits" (5ff07f6) and "starts a write's first request without waiting for a loop tick" (fc28071, now asserted with the worker's wait count); "uploads parts parked on another reactor's CreateMultipartUpload on their own reactor" replaces the mailbox case. New: the zero-body generator and a 72 MiB never-written part, the multipart ETag rule, 403 re-sign on UploadPart and final on Create, a lost conditional PUT then 412, a lost Complete recovered by HEAD and a NoSuchUpload that is not ours, a PUT torn mid-body retried on a new connection, part-buffer parking and the out_of_memory rule, device writes on every GPU, HTTPS writes on the user-space record layer (kTLS required skips on hosts without it), max_write_connections, expect_continue (honored and ignored), write stats, UploadParts starting in the Create's pass while a staged read is parked on a shared small pool (deadlocked before), a lost conditional answer of an earlier flush then 412, a lost Complete then a failed flush then NoSuchUpload, a Create on a stale pooled connection, writes and a flush beside a parked read, an ioctx destroyed before a half-written object, and a hidden throughput benchmark ([bench_rest_write]). test_rest_write_unimplemented.cpp is gone with the stubs.
…urce device Bugs of the converters that copy a GPU table's buffers one by one (GPU -> host_data_representation, GPU -> DISK and GPU -> GPU); they predate the io stack split (plan_column_copy and the fast GPU -> host converter come from NVIDIA#52, the GPU -> GPU column walk from NVIDIA#52 / NVIDIA#116, the GPU -> DISK converter from NVIDIA#96, view-backed gpu_table_representation from NVIDIA#116). - Sliced sources. A view-backed representation may hold a slice, but the serializers copy every buffer of a column and its children from the start, and plan_column_copy() / reconstruct_column_p2p() only asserted offset 0 in debug builds: in release builds a nullable slice starting at row 37 round-tripped as the table's leading rows with unshifted validity, and a prefix slice (offset 0) kept the extent of the table it was cut from in its children (DICTIONARY32 indices, STRING / LIST offsets, STRUCT fields), so it was restored with the wrong row count (GPU -> GPU failed with cudaErrorInvalidValue or "Invalid offsets column size" instead, or, for a fixed-width slice, silently returned the wrong rows). needs_compaction() detects both (non-zero offset anywhere, or a child extent that does not match its parent's rows), and serializable_view() compacts such a table first (cudf::table's deep copy), with the source device current, after the writer ordering; the copy lives until the transfer completed. GPU -> host and GPU -> DISK compact on the conversion stream; GPU -> GPU compacts on a stream of the source space and synchronizes it before the cross-device copies on the target stream read the copy. It needs device memory for the whole table on the source GPU; if that fails, the conversion fails and the batch stays where it is. - Failures after enqueued work. If the compaction (or anything later) throws after work reading the source was enqueued on the conversion stream, the stream is synchronized (drain_on_unwind, under the source-device guard) before the exception propagates, so the caller may release or mutate the source. GPU -> GPU does the same for its target stream: a failed allocation for column i+1 no longer returns while column i's peer copy is still queued (also before the compacted copy is freed), and the host-staged peer copy drains its stream before its pinned blocks go back to the pool. - Source device. GPU -> host_data made the source device current only around the compaction; it now keeps it current from the writer ordering through the copies and the release of the compacted copy. GPU -> DISK already does (and refuses a stream of another device); the compaction now runs in that scope too. Regressions [disk]: view-backed nested nullable slice [37, 2011) and prefix slices [0, 10) of a 100-row DICTIONARY32 / STRING / LIST / STRUCT table through host_data and DISK; a compaction whose second column allocation is refused (seam: a memory resource refusing large allocations) leaves the conversion stream idle when the error arrives, on both routes; [multi_gpu]: a GPU 1 table (also sliced) written late by a non-blocking stream spilled to DISK from GPU 0 on the default stream, a GPU 1 table converted to host_data from GPU 0 on a GPU 1 stream, nested / prefix / fixed-width slices converted GPU 0 -> GPU 1 and GPU 1 -> GPU 0 and back, and failed GPU -> GPU conversions (an allocation refused on the target, a compaction refused on the source) that leave the source's writer finished when the error arrives. The test sources that own a raw stream move it out of the moved-from object.
Add ioctx::remove_io_object_async / remove_io_object: the front end strips a file: scheme, invalidates the path in the ioctx's cache and metadata store before the request and again once it settled (as the write front ends do), and forwards to the new protected hook remove_io_object_async_io. The default hook resolves with not_implemented_error (errc::not_supported): the local backends do not implement it (DISK spaces keep removing their files with std::filesystem). rest_ioctx implements it with S3 DeleteObject on its control reactor (least backlogged, ties rotated): rest_reactor::delete_object_async posts a control_job of the new kind delete_object (DELETE /bucket/key, signed from its request spec through authorize_request, like AbortMultipartUpload), retried like the other control jobs with max_retry_attempts. Classification: 2xx is success; a 404 NoSuchKey, or a 404 without an error code whose error body was read to its end, is success too (the key is gone, also when the 404 arrives while the reactor stops); any other 404 (NoSuchBucket, ...) fails; a 404 whose error body was cut or not drained (the code cannot be known) is retried like a transport error (finish_control); a 403 is final. io_priority::low marks the job background: it starts only when no reads or writes wait, behind data work instead of ahead of it like blocking control calls (the reactor has no separate bulk class). The job owns the promise its on_done settles, and the intake / slot / retry queue own the job, so the returned future may be dropped without any reactor-owned bookkeeping. Before start() / after shutdown() (or from a reactor thread) it runs blocking on a private helper like head_object and returns a ready future. A delete queued or in flight at shutdown fails with operation_canceled (intake::close / drain). Stats: deletes (attempts sent). Tests: test/io/rest/test_rest_delete.cpp [rest][delete] -- 204 (presigned and header signing), 404 NoSuchKey and code-less 404 ok, 404 NoSuchBucket fails without a retry, a 404 whose (oversized) error body was not read is retried, 403 final, 503-then-204 retry, exhausted retries, unstarted reactor, dropped futures (incl. many low-priority ones) settle, shutdown with deletes in flight, non-s3 path, uring -> not_supported. The loopback server's status fault takes an optional body for the oversized 404.
Tier gains REMOTE, appended before SIZE (GPU 0, HOST 1, DISK 2 unchanged,
REMOTE 3, SIZE 3 -> 4). A REMOTE space keeps spilled batches as objects
under an object-store prefix; the core library stays io-free (it only
stores the io context pointer, as for DISK).
- memory/config.hpp: remote_memory_space_config {remote_id, memory_capacity
(default unbounded_remote_capacity = 1 PiB), uri_prefix, io_context},
thresholds as DISK (trigger = capacity, stop = 0.99 x capacity); appended to
the memory_space_config variant (existing indices unchanged); public
validate_remote_uri_prefix() (s3://bucket[/prefix]; rejects other schemes,
empty bucket, userinfo / port in the bucket, query / fragment, whitespace and
control characters, empty segments; returns it normalized).
ABI note: memory_space_config gains an alternative. It is appended, so the
existing alternatives keep their indices, but the variant's size and layout
change: code compiled against the old header must be rebuilt.
- memory_space: REMOTE constructor (reuses disk_access_limiter: capacity
bounds the bytes of reservations in flight, not the bytes stored), new
get_remote_uri_prefix(); get/set/get_or_init_io_context accept DISK or
REMOTE; to_string prints REMOTE.
- reservation_manager_configurator::add_remote_space(remote_id, uri_prefix,
capacity = unbounded_remote_capacity, io_context = nullptr); build() emits
REMOTE configs after the DISK ones.
- memory/disk_table.hpp: generate_remote_object_uri(prefix) ->
{prefix}/{hostname-pid-start_ms}/batch_{seq:016x}_{rand:08x}.cucascade (no
I/O), implemented in src/memory/remote_uri.cpp.
Tier audit (switches have `default:`, so the compiler does not flag missing
REMOTE cases; every use in the tree checked by hand):
changed here:
- memory/common.hpp Tier enum
- memory/config.hpp *_config::tier(), memory_space_config variant
- memory_space.cpp ctor overloads, get/set/get_or_init_io_context (DISK|REMOTE),
to_string switch (+REMOTE); get_disk_mount_path stays DISK-only
- reservation_manager_configurator build() (REMOTE after DISK)
- test/utils/mock_test_utils.hpp make_mock_memory_space (+REMOTE)
changed in the next commit (cudf layer):
- cudf/bandwidth_profiler.cpp canonical_type_for switch (default throws),
should_skip_pair DISK rules; test_bandwidth_profiler.cpp operator<< switch
- cudf/representation_converter_builtins.cpp registry (REMOTE converters)
no change needed (GPU/HOST-specific or tier-generic):
- memory_space.cpp dtor (HOST pool unregistration), make_reservation* /
get_available_memory / ... visits over the reservation allocator variant
- memory_reservation.hpp tier_memory_resource_trait (void, like DISK)
- memory_reservation_manager.{hpp,cpp} config visit and Tier-keyed lookups,
request strategies (any_memory_space_in_tier[s], *_to_downgrade/upgrade)
- memory/common.cpp make_default_allocator_for_tier (null resource branch)
- memory/topology_index.hpp GPU/HOST id extraction
- memory/fixed_size_host_memory_resource.cpp HOST space id
- data/data_batch.hpp GPU-only checks (:398, :677), data/common.hpp
- io/cache/types.cpp, io/datasource_factory.cpp HOST pool lookups
- cudf/host_data_representation.cpp HOST resource, bandwidth_profiler GPU
stream-owner checks, cudf/bandwidth_profiler.hpp GPU defaults
- benchmark/benchmark_{disk,representation}_converter.cpp (DISK/GPU/HOST only)
- tests enumerating explicit tier subsets (test_memory_reservation_manager,
test_data_*), test_disk_memory_space accessor rejection (GPU)
docs (AGENTS.md, README.md: "three tiers") are updated with the docs commit.
Tests: test/memory/test_remote_memory_space.cpp [memory][remote].
… context
HOST and GPU batches can now be converted to and from REMOTE memory spaces:
each batch becomes one object in an S3(-compatible) store.
- cucascade::remote_data_representation (core, io-free): the column layout
(disk_table_allocation, file_path = object URI) and an opaque handle to the
written object (detail::remote_object, defined by the cudf layer). clone() is
unsupported (as DISK).
- Object removal: releasing the handle never does I/O or blocks (it may run on
a reactor thread of its own context); it hands the context, the committed
write object and the URI to a process-wide removal queue. Its thread (never
a reactor thread) issues the DeleteObject (io_priority::low), owns the
context reference until the removal settled and releases it there: removals
survive the spaces / managers / representations that held their context (a
context's destruction would cancel them), and a context's last reference is
never dropped on its own reactor thread. The queue is created eagerly (by
make_remote_io_context and before every REMOTE write), entries are allocated
before the context is moved in (a failed allocation keeps the reference for
good rather than releasing it on the caller's thread) and moved by splicing,
and its thread never terminates on an exception. Failures are WARNed
(rate-limited) and left to the bucket's lifecycle rule. Each pass first
releases settled removals, then issues at most the ones queued when it
began, one at a time. Public drain_remote_object_removals() waits for all of
them (also run at exit; limits documented: late static releases, no
deadline, unbounded, no fork). The scoped, bounded
drain_remote_object_removals(contexts, timeout) is what embedders tear down
with: it waits only for the removals of the given contexts (counted per
context), for at most the timeout; at the deadline, removals still queued
for them are never issued, each context is shut down (settling its calls in
flight) and their entries are released on the calling thread, independent
of the queue's thread; it reports how many were abandoned (WARN; left to the
lifecycle rule) and whether it timed out.
- make_remote_io_context(remote_io_context_options) (cudf/remote_io_context.hpp):
a started REST context (no fs_cache, no default context) owning its pinned
staging (n_reactors x staging_bytes_per_reactor, 1 MiB blocks; the
base-from-member pattern of make_disk_io_context). Validates the endpoint
(exactly http(s)://host[:port]), region and keys (std::invalid_argument),
that the staging holds two parts, and that io_uring is usable
(std::system_error). Applies object_store CA/verify over rest.tls as the
registry does.
- io::rest::s3::make_authorizer(object_store_config) is now public and shared
by the io_context registry and make_remote_io_context.
- Converters HOST<->REMOTE and GPU<->REMOTE, factored with the DISK ones over
layout_{host,gpu}_batch / restore_{host,gpu}_batch (same 4 KiB-aligned
layout, metadata in memory). Writing (src/cudf/remote_io.cpp): a fresh name
(generate_remote_object_uri), open create_new, the slices plus zeros for the
alignment gaps so every part completes while the batch is written, flush =
commit (single PUT or CompleteMultipartUpload). Contexts from
make_remote_io_context write in windows of whole parts that fit half their
staging and admit concurrent conversions against it (exec::admission_control),
so a batch larger than the staging or concurrent spills never exhaust it.
Device reads are counted in staging blocks (every GET allocates whole
blocks; the GET count comes from the new public io::rest::max_read_gets(),
the reactor's segment planner's worst case: its smallest target,
max(1, n / min_segment_bytes, ceil(n / max_segment_bytes)), sharing the
count with physical_ranges; GETs are not cut at part boundaries),
so a wide table of tiny buffers is read in windows whose block count fits
half the staging, each admitted for exactly that. A conditional commit
refused with EEXIST, or one whose outcome the store left unknown (the new
public io::rest::may_be_commit_outcome_unknown: an attempt's answer was lost
and the retry got 412, or a lost Complete could not be verified; reported
as io_error, never EEXIST), renames once and retries; a key left by an
unknown outcome is not deleted (it could be another writer's) and is left
to the bucket's lifecycle rule. Any failure destroys the unflushed object
(the upload is aborted) and rethrows, leaving the source intact. Reads go
through the committed write object (known size, If-Match: no HEAD), with
bytes-delivered checks; a failed restore leaves the REMOTE representation
valid. A REMOTE space without an io context -> std::logic_error. GPU
sources: the source device is current from the writer ordering through the
write (a null stream resolves to that device's default stream for both; an
explicit stream of another device is refused, as for DISK), a
sliced view-backed source is compacted first (layout_gpu_batch, shared with
GPU -> DISK), and a failure after work reading the source was enqueued
drains the conversion stream before it propagates.
- bandwidth profiler: canonical type REMOTE -> remote_data_representation;
storage tiers (DISK, REMOTE) are never paired with each other, and REMOTE
pairs are measured only with the new bandwidth_profile_config::
measure_remote_pairs (default false).
Tests ([remote]): test/data/test_remote_converters.cpp (loopback S3 server:
round trips GPU and HOST of numeric, strings + nulls, lists/structs with nulls,
nullable, sliced, empty, single PUT, multi-part, larger than the staging, one
HTTPS case, view-backed sliced nested nullable source, GPU 1 source with a
late non-blocking writer converted from GPU 0 on the default stream
[multi_gpu] (also sliced), an explicit stream of another GPU refused before
any upload [multi_gpu], view-backed prefix slices (dictionary / string /
list / struct), a compaction failing part-way (stream drained), wide table of
tiny columns with slow GETs and 4 KiB segments (staging windows);
DeleteObject on destruction, removal outliving its space and manager
(throttled retry after the manager's teardown), release on a reactor thread of
the representation's own context, delete failure only warns; the scoped
drain: cancels a held removal at its deadline and releases the context, waits
only for its own contexts, never issues a removal still queued and does not
depend on the queue's thread (blocked in a removal through a shut-down
context), cancels nothing when settled in time, returns at once with a zero
timeout and nothing outstanding; persistent UploadPart 5xx and Complete 5xx -> throw +
abort + no object; 412 collision rename (PUT and Complete) and twice -> fail;
unknown commit outcome (lost PUT answer then 412; Complete NoSuchUpload not
verified) -> one retry under a new name;
GET failure on restore -> throw, representation intact; no io context ->
logic_error; concurrent spills over shared staging),
test/data/test_remote_io_context.cpp, and a REMOTE case in
test_bandwidth_profiler.cpp; shared helpers in test/utils/remote_test_utils.hpp;
max_read_gets against the reactor's GET count in test_rest_read_robustness.cpp
([rest]).
- docs/io/remote-tier-design.md: configuration (space prefix vs. io context, no default context, endpoint/credential validation), the memory_space_config variant change (appended alternative: indices kept, size/ABI changed), capacity semantics (in-flight bytes, no stored-bytes quota), object naming and layout, spill and restore steps (source device current from the writer ordering through the write, sliced sources compacted), staging windows (writes in whole parts, device reads counted in staging blocks) and admission, failure handling, lifecycle (the process-wide removal queue that owns contexts until their deletes settled, drain_remote_object_removals() and the scoped, bounded drain_remote_object_removals(contexts, timeout) embedders tear down with) and the requirement of an S3 lifecycle rule (expiration + AbortIncompleteMultipartUpload) for objects left by crashes or failed deletes; what is not implemented. - docs/io/rest-transport.md "Writes": DeleteObject (a control job on the reactor; 2xx, 404 NoSuchKey and a complete code-less 404 are success, other 404s fail, a cut 404 is retried; io_priority::low behind data work) and the `deletes` counter. remote-tier-design.md describes the segment planner's GET bound and the fresh-key retry after an unknown commit outcome. - README, docs/README, ARCHITECTURE.md (tier table and diagram), AGENTS.md, memory-management.md, bandwidth-profiler.md: the fourth tier.
aminaramoon
force-pushed
the
io/s3b-remote-space
branch
from
November 14, 2026 03:21
55bd545 to
d1baf92
Compare
giancao
added a commit
to giancao/cuCascade
that referenced
this pull request
Nov 14, 2026
Mark the places where NVIDIA#221 (S3 writes), NVIDIA#222 (object deletion, REMOTE tier) and NVIDIA#224 (NUMA reactor placement) would land, so the gaps are visible in the code. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
This adds a new
Tier::REMOTEmemory space that spills batches to S3 (or a compatible store) through the step-3a RESTio_context.ioctx::remove_io_object[_async]: a front end that stripsfile:and invalidates the path, plus a backend hook. REST issuesDELETE, where 204 and 404 both mean success; local backends returnnot_supported.docs/io/remote-tier-design.md, the architecture and memory docs, and AGENTS.Pre-existing fixes (
fix(cudf): serialize sliced GPU tables and order GPU spills on the source device):REMOTE tier (
feat(memory): add the REMOTE tier and remote memory spaces):Tier::REMOTEis appended beforeSIZE, so existing values are unchanged.remote_memory_space_config{remote_id, memory_capacity = 1 PiB, uri_prefix, io_context},add_remote_space,validate_remote_uri_prefixandgenerate_remote_object_uri.Tierwas audited by hand, because the switches havedefault:and the compiler won't flag a missing case. The list is in the commit message.memory_space_configvariant gains an appended alternative, so its size changes.Spill and restore (
feat(cudf): spill to and restore from REMOTE spaces through a REST io context):make_remote_io_contextowns pinned staging, has no fs_cache, and has no default context, because credentials are required.remote_data_representation.create_newwithIf-None-Match, retried under a new name on 412; windowed vectored writes; commit on flush; abort on failure.If-Match; reads are windowed within the staging, which is counted per GET.drain_remote_object_removals()before CUDA teardown.Semantics (MVP)
{prefix}/{run_id}/. A bucket lifecycle rule is required (expire the prefix plus AbortIncompleteMultipartUpload).Review fixes included
Two review rounds, a deep review and a design review, produced these fixes:
Tests
All tests use loopback S3/HTTPS servers.
cucascade_tests [remote]cucascade_io_tests [delete]cucascade_cudf_tests [remote][disk][multi_gpu][fast],[gpu_data_representation][rest]pixi run testpasses 10/10.Review round
This PR went through a multi-agent review:
Every fix below has a regression test that fails with the fix reverted. Review artifacts:
review/<PR>/(findings, TRIAGE.md, PLAN.md, FINAL.md).21 findings: 9 valid, 1 major.
Fixes, folded into this PR's commits:
NoSuchKey.NoSuchBuckethad been treated as success.drain_remote_object_removals(contexts, timeout), which Sirius #2381 uses so that teardown can't block for a whole retry budget.The final review's major ("a throwing
shutdown()") can't happen, becauseshutdown()isnoexcept. The drain was hardened anyway, with astatic_assertand an RAII unlink.Rebase note (REST re-split)
Rebased onto the re-split #221, whose S3 writes now run on the
uring_ultra-design reactor of #227. Re-done on the new reactor: S3 DeleteObject is acontrol_job(least-backlogged reactor; background priority; the 404 rule:NoSuchKey, or a code-less 404 with a complete body, is success; a 404 whose body was cut is retried);max_read_getsis the new segment planner's worst casemax(1, n/min_segment, ceil(n/max_segment)); the fresh-key retry of the REMOTE converter now also covers the newio_error"commit outcome unknown" (acreate_newcommit whose first attempt lost its answer and whose retry got 412), throughio::rest::may_be_commit_outcome_unknown. New converter tests: a lost PUT answer followed by 412 retries under a new name; a lost Complete answer recovered through the HEAD check uses the one key it wrote; a Complete answeredNoSuchUploadwithout a lost answer fails the spill with no leftover key.Rebase onto
main(2026-11-06)Rebased with the whole stack onto
main5489112: #218 was squash-merged as394465c(with its 20 review-round commits that this stack did not have yet), followed by #212 (ORDERED/UNORDERED repositories) and #240 (pool construction cleanup, one-shot future terminal,list_entry.etag). It was then rebased again ontomaina11ff66, which adds #236 (atomics.hppincludes what it uses) and #232 (PR label checker off), with no conflicts. One conflict in this PR's commits: the REMOTE branch ofmake_mock_memory_spacenext to the DISK branch (see #220).Verified at this branch's
5489112tip (Release,sm_89,CUCASCADE_WARNINGS_AS_ERRORS=ON, libcudf 26.12 nightly (00a724), 2× RTX 6000 Ada):cucascade_tests170 cases,cucascade_io_tests746 (1 skipped:rest_ioctx over HTTPS with kTLS required fails without tls.ko, as this host has notls.ko),cucascade_cudf_tests336, all passing; the benchmarks build. Thea11ff66step changes onlymain's two files; the bottom (#227) and top (#224) of the stack were rebuilt and retested on it with the same counts.Rebase onto
main(2026-11-13)Rebased onto the rebased #221 (
d3f429a, onmain3db0c56; previous head55bd545). Sirius companion pin:d1baf92.Conflicts, in
fix(cudf): serialize sliced GPU tables…:mainrestructured the peer-copy fallback (peer_copy_into_async) andconvert_gpu_to_gpu, each now draining its stream on an exception. The sliced-table compaction (needs_compaction,serializable_view) anddrain_on_unwindare kept for the host and DISK converters. Inconvert_gpu_to_gputhe compaction runs insidemain'stry, with the compacted copy declared before it so it outlives the drainingcatch. The host-staged copy paths keepmain's own drains, which already cover what this commit added there. Also: an include list in the REMOTE converters commit, and theAGENTS.mdchanges of the docs commit, ported tomain's rewritten file (four tiers, and the note that tier switches havedefault:branches).Verified on 2× RTX 6000 Ada in the
cuda-13-stableenv (Release,-Werror, CUDA 13.2, gcc 15.3, libcudf 26.10.02): a fullcmake --preset releasebuild (libraries, tests and benchmarks) has 0 errors andctestpasses 11/11; the hidden multi-GPU tests ([multi_gpu],[multi-device]) run 14 cases, 12 passed and 2 skipped (this host has no direct peer DMA), and include this PR's sliced-table GPU→GPU regressions. The pinned pre-commit hooks pass.