Skip to content

feat(memory): REMOTE object-store memory tier (S3 spill via the native REST io_context) [stacked on #221] - #222

Draft
aminaramoon wants to merge 26 commits into
NVIDIA:mainfrom
aminaramoon:io/s3b-remote-space
Draft

aminaramoon wants to merge 26 commits into
NVIDIA:mainfrom
aminaramoon:io/s3b-remote-space

Conversation

@aminaramoon

@aminaramoon aminaramoon commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Draft · stacked PR 6 of the IO split of #216 (step 3b). Base: #221 (S3 writes), which sits on #220, #219 and then main.
Review only the 5 commits after d3f429a: aminaramoon/cucs@io/s3-rest-uring...io/s3b-remote-space

Stack (re-ordered): #209 (merged) → #218 (merged) → #227 (merged) → #219 (GDS read) → #220 (write API + DISK via io_context) → #221 (S3 writes) → this (3b: REMOTE object-store memory space) → #224 (NUMA placement).
Sirius companion: sirius-db/sirius#2381.

What

This adds a new Tier::REMOTE memory space that spills batches to S3 (or a compatible store) through the step-3a REST io_context.

Commit Summary
fix(cudf): serialize sliced GPU tables and order GPU spills on the source device Pre-existing bugs that predate the io split. See the list below this table.
feat(io): delete objects through ioctx; S3 DeleteObject Adds ioctx::remove_io_object[_async]: a front end that strips file: and invalidates the path, plus a backend hook. REST issues DELETE, where 204 and 404 both mean success; local backends return not_supported.
feat(memory): add the REMOTE tier and remote memory spaces See the list below this table.
feat(cudf): spill to and restore from REMOTE spaces through a REST io context See the list below this table.
docs(memory): document the REMOTE tier docs/io/remote-tier-design.md, the architecture and memory docs, and AGENTS.

Pre-existing fixes (fix(cudf): serialize sliced GPU tables and order GPU spills on the source device):

  • Sliced and prefix-sliced GPU views were serialized with unshifted null masks and offsets. This affected STRING/LIST/STRUCT/DICTIONARY columns, so a restore could produce wrong validity or extra rows.
  • The source GPU is now current from the writer wait through submission for the host_data, DISK and REMOTE paths.
  • A failed compaction now drains its stream before the error propagates.

REMOTE tier (feat(memory): add the REMOTE tier and remote memory spaces):

  • Tier::REMOTE is appended before SIZE, so existing values are unchanged.
  • Adds remote_memory_space_config{remote_id, memory_capacity = 1 PiB, uri_prefix, io_context}, add_remote_space, validate_remote_uri_prefix and generate_remote_object_uri.
  • The core library stays io-free.
  • Every use of Tier was audited by hand, because the switches have default: and the compiler won't flag a missing case. The list is in the commit message.
  • ABI: the memory_space_config variant gains an appended alternative, so its size changes.

Spill and restore (feat(cudf): spill to and restore from REMOTE spaces through a REST io context):

  • make_remote_io_context owns pinned staging, has no fs_cache, and has no default context, because credentials are required.
  • Adds remote_data_representation.
  • Adds HOST/GPU↔REMOTE converters:
    • Write path: create_new with If-None-Match, retried under a new name on 412; windowed vectored writes; commit on flush; abort on failure.
    • Read path: read-back with If-Match; reads are windowed within the staging, which is counted per GET.
  • A process-wide off-reactor removal queue issues the DeleteObject when a representation dies. It never blocks a reactor thread and survives teardown of the space or manager. Embedders call drain_remote_object_removals() before CUDA teardown.

Semantics (MVP)

  • Capacity bounds spills in progress, the same as DISK. There is no stored-bytes quota.
  • REMOTE is the last tier; nothing downgrades from it.
  • Credentials come from the io context; nothing is read from the environment.
  • Leftovers from crashes or failed deletes stay under {prefix}/{run_id}/. A bucket lifecycle rule is required (expire the prefix plus AbortIncompleteMultipartUpload).
  • Not included: DISK↔REMOTE converters.

Review fixes included

Two review rounds, a deep review and a design review, produced these fixes:

  • Sliced, prefix-sliced and dictionary views.
  • Source-device guard ordering for DISK, REMOTE and host_data.
  • Read-staging undercount for wide tables and for odd segment bounds.
  • Last-owner delete: the old version could deadlock on a reactor or be cancelled at teardown; this led to the removal queue.
  • Allocation-failure handling in the queue.
  • Docs updated to four tiers.

Tests

All tests use loopback S3/HTTPS servers.

Tag Result
cucascade_tests [remote] 12
cucascade_io_tests [delete] 10
cucascade_cudf_tests [remote] 34
[disk] 67
[multi_gpu] 5 (4 GPUs)
[fast],[gpu_data_representation] 57
[rest] 389 pass, 11 skip (kTLS/DEFER_TASKRUN not available here)
  • Remote coverage: the round trips cover nested, sliced and empty tables, multipart and single PUT, and HTTPS.
  • Failure cases: UploadPart and Complete faults, 412 collisions, GET failure on restore, and destruction from a reactor callback.
  • Full suite: pixi run test passes 10/10.
  • Builds: every commit builds, as does the cudf-free build.
  • Lint: clean.

Review round

This PR went through a multi-agent review:

  1. Adversarial "unhappy-path" reviewers (Grok 4.7): one per commit, one for the whole PR, plus topic experts.
  2. Triage of every finding for merit and reproducibility (Sol 6.1 xhigh, or Claude Fable 5.1 where a content filter blocked it).
  3. Fix plans (Opus 5.5), then fixes, then a final correctness review.

Every fix below has a regression test that fails with the fix reverted. Review artifacts: review/<PR>/ (findings, TRIAGE.md, PLAN.md, FINAL.md).

21 findings: 9 valid, 1 major.

Fixes, folded into this PR's commits:

  • GPU→GPU copies of sliced tables. Some slice shapes silently produced wrong rows or null masks.
  • DeleteObject: a 404 counts as success only for NoSuchKey. NoSuchBucket had been treated as success.
  • A new scoped, deadline-bounded drain_remote_object_removals(contexts, timeout), which Sirius #2381 uses so that teardown can't block for a whole retry budget.
  • The stream/device check applied to the GPU→REMOTE converter.
  • Docs.

The final review's major ("a throwing shutdown()") can't happen, because shutdown() is noexcept. The drain was hardened anyway, with a static_assert and an RAII unlink.

Rebase note (REST re-split)

Rebased onto the re-split #221, whose S3 writes now run on the uring_ultra-design reactor of #227. Re-done on the new reactor: S3 DeleteObject is a control_job (least-backlogged reactor; background priority; the 404 rule: NoSuchKey, or a code-less 404 with a complete body, is success; a 404 whose body was cut is retried); max_read_gets is the new segment planner's worst case max(1, n/min_segment, ceil(n/max_segment)); the fresh-key retry of the REMOTE converter now also covers the new io_error "commit outcome unknown" (a create_new commit whose first attempt lost its answer and whose retry got 412), through io::rest::may_be_commit_outcome_unknown. New converter tests: a lost PUT answer followed by 412 retries under a new name; a lost Complete answer recovered through the HEAD check uses the one key it wrote; a Complete answered NoSuchUpload without a lost answer fails the spill with no leftover key.

Rebase onto main (2026-11-06)

Rebased with the whole stack onto main 5489112: #218 was squash-merged as 394465c (with its 20 review-round commits that this stack did not have yet), followed by #212 (ORDERED/UNORDERED repositories) and #240 (pool construction cleanup, one-shot future terminal, list_entry.etag). It was then rebased again onto main a11ff66, which adds #236 (atomics.hpp includes what it uses) and #232 (PR label checker off), with no conflicts. One conflict in this PR's commits: the REMOTE branch of make_mock_memory_space next to the DISK branch (see #220).

Verified at this branch's 5489112 tip (Release, sm_89, CUCASCADE_WARNINGS_AS_ERRORS=ON, libcudf 26.12 nightly (00a724), 2× RTX 6000 Ada): cucascade_tests 170 cases, cucascade_io_tests 746 (1 skipped: rest_ioctx over HTTPS with kTLS required fails without tls.ko, as this host has no tls.ko), cucascade_cudf_tests 336, all passing; the benchmarks build. The a11ff66 step changes only main's two files; the bottom (#227) and top (#224) of the stack were rebuilt and retested on it with the same counts.

Rebase onto main (2026-11-13)

Rebased onto the rebased #221 (d3f429a, on main 3db0c56; previous head 55bd545). Sirius companion pin: d1baf92.

Conflicts, in fix(cudf): serialize sliced GPU tables…: main restructured the peer-copy fallback (peer_copy_into_async) and convert_gpu_to_gpu, each now draining its stream on an exception. The sliced-table compaction (needs_compaction, serializable_view) and drain_on_unwind are kept for the host and DISK converters. In convert_gpu_to_gpu the compaction runs inside main's try, with the compacted copy declared before it so it outlives the draining catch. The host-staged copy paths keep main's own drains, which already cover what this commit added there. Also: an include list in the REMOTE converters commit, and the AGENTS.md changes of the docs commit, ported to main's rewritten file (four tiers, and the note that tier switches have default: branches).

Verified on 2× RTX 6000 Ada in the cuda-13-stable env (Release, -Werror, CUDA 13.2, gcc 15.3, libcudf 26.10.02): a full cmake --preset release build (libraries, tests and benchmarks) has 0 errors and ctest passes 11/11; the hidden multi-GPU tests ([multi_gpu],[multi-device]) run 14 cases, 12 passed and 2 skipped (this host has no direct peer DMA), and include this PR's sliced-table GPU→GPU regressions. The pinned pre-commit hooks pass.

@copy-pr-bot

copy-pr-bot Bot commented Oct 28, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@mbrobbel

Copy link
Copy Markdown
Member

/ok to test 3b8b1d1

@aminaramoon
aminaramoon force-pushed the io/s3b-remote-space branch 2 times, most recently from 13bd3e3 to 75417e7 Compare October 29, 2026 05:38
@mbrobbel

Copy link
Copy Markdown
Member

/ok to test 75417e7

@mbrobbel

Copy link
Copy Markdown
Member

/ok to test 65b46f7

@mbrobbel

mbrobbel commented Nov 2, 2026

Copy link
Copy Markdown
Member

/ok to test f3949dc

@aminaramoon
aminaramoon force-pushed the io/s3b-remote-space branch 4 times, most recently from f941c00 to 55bd545 Compare November 12, 2026 16:47
mike-wendt pushed a commit that referenced this pull request Nov 12, 2026
…LS; drop libcurl and kvikIO (IO split step 0b) (#227)

> **Draft · stacked PR 2 of the IO split of #216 (step 0b).** Base:
`main`.
> **Review:** the single squashed commit on this branch.
>
> Stack (re-ordered): #209 (merged) → #218 (merged) → **this (0b:
io_uring REST read + TLS)** → #219 (GDS read) → #220 (write API + DISK
via io_context) → #221 (S3 writes) → #222 (REMOTE tier) → #224 (NUMA
placement).

## Why this PR exists, and why it moved
#221 used to carry the io_uring REST transport, the TLS layer, S3 writes
and the kvikIO removal, late in the stack. A scan-replay benchmark of
TPC-H SF1000 on S3 (`tpch_scan_bench`) showed the simpler S3 reactor
beating that design: all 22 queries 59 s vs 80 s, q1 io-only 15.8 vs 9.8
GiB/s, up to 36 GiB/s with `min_mss` and 8 pinned reactors. So the READ
half moves here, right after parity; S3 writes stay in #221.

## What

**Transport (`src/io/rest/net`).** One io_uring per reactor, one
keep-alive connection per slot; at most one primary + one cancel SQE per
slot; slot never reused before both CQEs are reaped. Connects over a
resolver thread's addresses (DNS never blocks the loop). Body bytes of a
data GET land **directly into the destination iovecs** (pinned staging,
host memory or `fs_cache` chunks); headers go through a scratch buffer.

**TLS (`src/io/tls`, `include/cucascade/io/tls`).** `tls::config`
(version bounds, `ktls = prefer | require | disable`, user-space RX via
a built-in record layer or OpenSSL, io-wq placement, trust-store
discovery, generation-safe session cache). After the OpenSSL handshake,
installs kTLS RX+TX when policy and kernel allow (plain
`SEND`/`RECVMSG`; tickets skipped; TLS 1.3 KeyUpdate re-keyed where the
kernel can, else retire + retry); otherwise the built-in user-space
record layer decrypts the ciphertext ring straight into the destination.
`ktls = require` fails with `errc::ktls_unavailable`.

**HTTP/1.1 codec (`src/io/rest/http`).** Zero-copy, allocation-free
response parser, chunked body decoder, `Content-Range` value parser,
per-field S3-focused dispatch table, iovec sink and a retry/reuse
classifier. Request-head writer emits a single contiguous SEND and
records field spans for the SigV4 signer. Chunked and close-delimited
GET bodies are reported as `unsupported_framing` instead of being
silently coalesced.

**Reactor and pool (`src/io/rest`).** Adaptive physical GETs (`backlog /
free connections` clamped to `[min_segment_bytes 4 MiB,
max_segment_bytes 16 MiB]`), retry heap with backoff, jitter and
Retry-After, bounded 403 re-signs, Content-Range validation, "server
ignored Range", short-read retry, demand before prefetch. Device
destinations stage through the HOST memory resource blocks and copy on
the slice's stream with the slice's device current; the slot is held
until the copy completes. Staging exhaustion **parks** (leases are
counted per reactor; a read fails with `out_of_memory` only when the
reactor holds no staging).

**Control plane** (HEAD, footer probe, LIST, `resolve_footer_objects`)
runs on the pool's own connections: least-backlogged reactor, at most
half of a reactor's connections while data waits; never on the caller's
thread (a private helper serves calls made before `start()` or from a
reactor thread).

**Wait modes.** `epoll_pwait2` over {eventfd, retry timerfd, ring fd}
(default) or `io_uring_submit_and_wait_timeout` (`wait_mode = uring`,
with `defer_taskrun` / `single_issuer`).

**Generic exchange.** An optional request body (iovecs or generator) and
`Expect: 100-continue` are already supported and tested against a
loopback echo. This is the hook #221 (S3 writes) builds on; no S3 write
semantics here.

### Pool behaviour: parity with the libcurl stack it replaces
Checked against the curl source (`conncache.c`, `cf-socket.c`): a pooled
connection is reused only if idle for less than `conn_max_age` (**20
s**, as `CURLOPT_MAXAGE_CONN`; #221 had silently moved this to 4 s) and
it looks alive (zero-timeout poll: input pending or HUP with no response
outstanding = dead). Dead or too-old connections are replaced **lazily,
one slot at a time**, when a request needs them: there is never a mass
reconnect. A request that dies on a reused connection before any
response byte is retried once for free on a **fresh** connection.
`warmup(bucket)` stays rate-limited to once per `conn_max_age` per
bucket and pre-opens up to `max_connections` per reactor with real `LIST
max-keys=0` exchanges on fresh connections. TCP keepalive 60/60 and
`TCP_NODELAY` on. curl's upkeep is a no-op for HTTP/1.1, so nothing is
sent while idle by default; `rest.upkeep_http1` (off) sends staggered
zero-key LIST exchanges on idle connections every `upkeep_interval`
instead.

### NIC-driven reactor sizing and pinning
`rest_n_reactors` is a **total** (never per GPU). With `0`, the egress
NIC (`rest.interface`, else the default route) is inspected: when its
queue IRQs are steered to a strict subset of the NIC's NUMA node's cores
(requested **and** effective masks narrow), the pool gets one reactor
per IRQ CPU, each pinned to it before it creates its ring and raised to
the highest priority the process may set (SCHED_FIFO if permitted, else
nice; never widening the inherited affinity). Otherwise: the configured
count (default 2), unpinned. The reactor's io-wq is kept off the pool's
IRQ CPUs. Startup log reports the plan; `planned_reactor_cpus()` exposes
it for test/introspection.

### `min_mtu`
With jumbo frames, some pool connections negotiate small segments, which
costs decryption time per byte. After the TCP connect, the effective MTU
= `TCP_MAXSEG` + 40/60 (+12 with TCP timestamps) is compared with
`rest.min_mtu`; too small → RST and reconnect on the next address, at
most `mtu_reject_max_attempts` (8) times per request, then accept
(`mtu_fallbacks`). Every close uses `SO_LINGER{on, 0}`
(`close_with_rst`, default on) so no socket lingers in TIME_WAIT /
FIN_WAIT. Stats: `mtu_rejects`, `mtu_fallbacks`, `mss_min`, `mss_max`.

### Removed
- **kvikIO backend** (`refactor(io)!`): config, backend, tests and docs;
the datasource registry has no catch-all any more (unclaimed paths
return "no backend").
- **libcurl**: no longer a dependency of the REST path.

## Config
Every #218 field keeps its name and default (`max_connections 64`,
`conn_max_age 20 s`, `upkeep_interval 15 s`, `request_timeout_s 30`,
stall detector 64 KiB/s × 30 s, `footer_probe_bytes 512 KiB`, list caps,
retries). `ca_bundle_path` / `tls_verify` fill the new `tls` block. New:
`tls`, `min/max_segment_bytes`, `receive_buffer_bytes`,
`max_recv_iovecs`, `recv_waitall`, `queue_depth`, `sqpoll`,
`single_issuer`, `defer_taskrun`, `register_ring_fd`,
`iowq_max_unbound/bound`, `wait_mode`, `copy_poll_interval`,
`hold_slot_during_copy`, `dns_ttl` / `accumulate` / `max_addresses`,
`connect_timeout`, `keepalive_*`, `so_rcvbuf` / `sndbuf`,
`busy_poll_us`, `tcp_user_timeout_ms`, `upkeep_http1`, `min_mtu`,
`mtu_reject_max_attempts`, `close_with_rst`, `fan_out`, `interface`,
`pin_to_irq_cpus`; `rest_n_reactors = 0` means auto.
`config::validate()` rejects invalid values (including
`max_retry_attempts` / `max_auth_retry_attempts` of 0, which used to be
clamped silently).

⚠️ API notes: `reactor_context` is no longer copyable (it holds a mutex;
`make_shared` users are unaffected); `rest_ioctx::stats()` is new;
libcurl is no longer a dependency.

## Behaviour differences from #221 that an embedder could notice
A cut body is retried whole, not resumed; a 200 for a sub-range fails as
"server ignored Range"; no link estimator/controller; the #221 fields
`keep_warm_connections`, `enable_controller`, `link_gbps`,
`ring_entries` (→ `queue_depth`), `tls_handshake_timeout` (→
`connect_timeout`), `write_split_bytes`, `expect_continue` do not exist
here (the last two return with #221).

## Review round
Two independent reviews (transport/TLS/pool;
scheduler/destinations/contract/NIC) with mandatory reproduction: 0
blockers. Fixed with regression tests: staging exhaustion parked instead
of failing (reproduced OOM on a 2 MiB pool); `upkeep_http1` busy-spin
while a GET was in flight (1.9 M passes/s → < 100); control jobs no
longer pinned to reactor 0 nor starving data (a footer batch delayed a 4
KiB read by 3 s → < 400 ms); `_inflight` leak on a throwing submit;
control call from a reactor thread deadlocked; stale retry now goes to a
fresh connection; CQEs reaped before deadline checks; kTLS KeyUpdate
reply `EWOULDBLOCK` no longer fatal; idle drain loops while readable;
pins re-intersected with the worker's affinity; pool plan kept per
ioctx. Verified clean by the reviewers: coordinator / `queued_bytes`
accounting, multi-GPU staging on 4 GPUs, the #218 public contract (base
REST tests byte-identical and passing), a 400-exchange fault-injection
run in all three ring modes.

## Tests
`pixi run test` 10 / 10; 262 REST/TLS cases (8 skip: kTLS without
`tls.ko`); 257 cudf cache tests through the REST backend; cudf-free
build links without curl; 0 compiler warnings; lint clean. Loopback
smoke (user-space TLS, 1 GiB in 16 MiB GETs, 2 reactors × 16
connections): 2.8–3.1 GiB/s. **Not tested here:** kTLS data path (no
`tls.ko`; unit-tested with fakes), real S3, jumbo-frame MSS negotiation
(the probe is tested through a seam), a NIC with steered IRQs (this
host's are not steered; fake sysfs trees cover both cases).

## Branch state
The 16 development commits have been **squashed into one** against the
PR base. Verified at the squashed tip (Release, `sm_89`,
`CUCASCADE_WARNINGS_AS_ERRORS=ON`, libcudf 26.12 nightly, 2× RTX 6000
Ada): `cucascade_tests` 166 cases, `cucascade_io_tests` 511 (1 skipped:
`rest_ioctx over HTTPS with kTLS required fails without tls.ko`, as this
host has no `tls.ko`), `cucascade_cudf_tests` 263, all passing; the
benchmarks build.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Add the private cufile_api dlopen loader (libcufile.so.0, RTLD_NODELETE,
read entry points only) and the process-wide cuFile platform probe
(probe_platform / gds_available).  The cuFile backend never links
libcufile: it needs only cufile.h at configure time.  find_path(cufile.h)
gates the sources, the tests and the propagated CUCASCADE_HAS_CUFILE
definition, independently of CUCASCADE_BUILD_CUDF, so a cudf-free build
keeps the backend.

The loader resolves read entry points only; write, buffer-registration and
stream entry points are left out.
Add cufile_reactor / cufile_ioctx, a read-only GPUDirect Storage backend:

- device slices of at least gds_threshold bytes of a registered file are
  read by synchronous cuFileRead calls of at most max_op_bytes (64 MiB),
  split on 4 KiB file boundaries with the EOF tail read on its own; short,
  zero and oversized returns are errors, a failing cuFileRead degrades to
  POSIX.  Destinations are never registered (cuFileBufRegister): every
  read passes the exact device pointer.
- everything else goes through POSIX: host-only reads O_DIRECT (staged
  through the scratch when unaligned), host+device pairs pread + H2D
  copy, device-only reads double-buffered through two 8 MiB pinned scratch
  slots on a reactor-owned non-blocking copy stream.
- a device slice whose stream still has queued work is parked behind an
  event instead of blocking the worker; a parked slice of a request that
  already failed settles at once instead of waiting for its stream, and
  shutdown cancels parked slices and returns the pinned scratch to a
  process-wide pool rather than calling cudaFreeHost (no device
  synchronization).
- a read whose H2D copy was enqueued but whose event could not be recorded
  waits for the copy stream before settling, so neither the destination
  nor the scratch is released under a running copy.
- strict priority with bounded low service, per-pass byte quantum.
- opens reopen the buffered fd for O_DIRECT through /proc/self/fd, check
  dev/inode before registering (retry once, then descriptor_mismatch_error)
  and carry the buffered fd's local_stat identity and size.
- cufile_ioctx fans a vectored read with more than two slices out over
  every reactor of its count-based pool.

Routing: io_backend::cufile and io_context_type::cufile;
io_config::{cufile_n_reactors, cufile} under CUCASCADE_HAS_CUFILE.  With
backend=cufile, local regular files resolve to cuFile and s3:// stays on
REST; native routing is unchanged, and an explicit cufile
request in a build without cuFile is refused instead of served by another
backend.  The reactor traits opt out of the prefetching cache.  Vectored
device-read signatures are unchanged; their completion contract is now
documented at ioctx::device_readv_async_io and
datasource::device_read_ranges_async.
Cover read scheduling: another stream's read and a host read are served
while a device slice waits for its held stream, shutdown cancels a parked
slice promptly on the cuFile and the staged path (with pooled scratch
already handed out), reads submitted before start are cancelled, and an
active low request is parked for a high one with strict or bounded low
service.

Add the hidden read-only throughput probe [.cufile-bench]
(CUCASCADE_IO_BENCH_FILE/BYTES/ROUNDS/REACTORS/BACKEND/ONLY/GDS_THRESHOLD).
It validates that the operator-supplied input exists and is large enough
and never creates, resizes or syncs it; the raw cuFile submission and
write probes stay out.
Add topology_discovery::discover_storage_for_path and
discover_storage_for_descriptor, returning a path_storage_info snapshot:
the opened file's resolved path and device numbers, its mount (statx mount
ID when the kernel reports it, else the longest component-boundary mount
point of the file's device, with bind mounts and shadowed mounts handled
and escaped mountinfo fields decoded), filesystem type / source / options,
and for block filesystems the leaf disks reached through partition parents
and device-mapper / md slaves (cycle-protected), each with drive class,
PCI bus ID and NUMA node.

Network and memory filesystems are classified without inventing devices;
overlay / FUSE / pseudo filesystems, unreadable procfs or sysfs and
unresolvable stacks are reported as unknown, which is distinct from
missing, non-regular and inaccessible paths.  Discovery is passive
(procfs / sysfs only, no CUDA, no cuFile, no writes), never throws and
caches nothing.  It lives in its own source of the topology-only library;
fixture procfs / sysfs roots are injected through a src-private header
used by the tests only.
Add cufile::probe_path(path, config) and probe_object(object, config), and
cufile_ioctx::path_capability(io_object) for objects it opened.  They
combine the effective compat / use_odirect policy, the process-wide
platform probe, passive storage discovery of the opened descriptor
(topology_discovery::discover_storage_for_descriptor, so the verdict
describes the file actually read) and the open's O_DIRECT / registration
outcome into a path_info with every piece of evidence and one verdict:
unavailable, compatibility_only, direct_candidate or unknown, with a
reason.

Only the dstatusflags bit of the file's own storage class counts (NVMe,
NVMe-oF, NFS, Lustre, WekaFS, GPFS, BeeGFS, VirtioFS), never any other
bit; ext4 (not data=journal) and xfs are the supported local
filesystems.  A direct candidate is not a DMA guarantee.  Unknown stacks
and failed property queries stay unknown; compat OFF refusals are
reported as unavailable with their cause.  probe_path is explicitly
active (it opens the cuFile driver and registers the file, reads nothing,
writes nothing); verdicts are logged once per device and verdict.

platform_info now distinguishes an opened driver whose properties could
not be read (properties_available) from a compatibility-only one;
gds_available() requires readable properties.
Document path_storage_status and the storage stack discovery that
cufile::probe_path builds its per-file GDS verdict on.
Add the ioctx write API: write_mode / write_open_options / write_slice,
open_io_object_for_write, synchronous host_write, asynchronous host / device
/ vectored / mixed writes, flush and flush_async, supports_write and
supports_device_write, and the backend hooks behind them (defaults report
std::errc::not_supported).  Asynchronous forms are noexcept and report every
error through the future.

templated_ioctx gains the optional io_write_reactor_c contract with separate
reactor_write_traits, so non-writing reactors keep compiling unchanged.  Read
and write dispatch share dispatch_slices; reads keep their batching, callback
and priority behaviour.  Write slices are validated (null, overlap, overflow,
past off_t), never clamped to EOF, and cut at granule-aligned multiples of the
reactor config's write_split_bytes.  io_op_request::finish_success raises a
writable object's size before the coordinator settles.

A write invalidates its path in the writing ioctx's fs_cache and metadata
store before dispatch and again before success or failure is visible (the
completion is fully allocated before the write is dispatched, so nothing can
fail between starting physical work and attaching the second pass); a
truncating or replacing open invalidates too.  fs_cache retires the path's
generations and never hands an invalidated generation out again; a
pending_invalidation holds an in-flight slot until it fires, so
shutdown_cache waits for it.  Retiring the generations is allocation-free and
happens first, so an allocation failure can only defer the eager reclaim of
their idle chunks, never leave a generation reusable.  metadata_store gains a fixed 256-slot
per-path-hash write epoch: objects are stamped at open, pre-write opens can
neither observe nor publish, loads never join across a write, and entries are
returned only to opens of the epoch they were published under.
invalidate_written retires every profile of the path; targeted invalidate
leaves the epoch alone.  Writable objects carry an opened_handle identity.
Add io::not_implemented_error, a std::system_error with
std::errc::not_supported whose message names the backend and operation and
says "not yet implemented".  The ioctx write front ends report it when the
backend cannot write at all (a host-only writer asked for a device source
still gets a plain not_supported), and so do the default write hooks.

The REST context stays read-only and reports it from explicit write, flush
and write-open hooks: synchronous hooks throw, asynchronous ones return a
failed future, with no network request.  The REST reactor gains static
write-open / flush and member host_write stubs, and its enqueue fails write
and flush slices instead of reading them as GETs.  Because the stubs satisfy
io_write_reactor_c structurally, reactor_write_traits<rest_reactor> is
specialized to false/false next to the reactor, before any context
instantiation.

Tests cover every entry point through ioctx&, the direct hooks and direct
reactor submission: exception type, code and message, no network side
effects, and zero-size no-ops.
…e read

start_device_copy enqueued the staged H2D copy and its completion event in
one call.  When copy_async failed part-way, or only the event record failed,
the slot was settled at once: its staging block went back to the pool and
the request resolved while the copy might still read the block and write the
caller's destination.

Record the event separately and, on any failure after the copy may have been
queued, synchronize the caller's stream before settling, as the cuFile
reactor already does.  A process-wide test seam
(uring::detail::set_copy_event_record_failure_for_testing) makes the record
fail; the regression holds the caller's stream so the copy cannot have run,
and checks the destination holds the file's bytes once the failed read
settled.
A read future resolves with its coordinator's bytes_requested, the bytes of
its slices as clamped at dispatch to the size recorded on the io_object.  That
count is the bytes delivered only because every reactor path fails a transfer
that comes up short of the clamped range instead of settling it as a success:
uring resubmits short reads and fails a zero-byte one with EIO (also when its
O_DIRECT range was widened), and the cuFile reactor fails a short cuFileRead,
staged pread or host pread with EIO.  Nothing pinned that contract.

Document it on grouped_coordinator and device_readv_async_io, and test it
for uring (O_DIRECT and buffered) and both cuFile read paths: reads running
past EOF resolve with exactly the bytes up to EOF (scalar and vector forms,
bytes checked), a read starting at EOF resolves 0, and reads of a file
truncated after it was opened fail with std::errc::io_error for device and
asynchronous host reads, while a part the file still holds reads in full and
the synchronous host read reports what pread delivered.
cudaStreamPerThread names the per-thread default stream of whichever thread
uses the handle.  The reactors interpret a device slice's stream on their
worker thread: the uring reactor enqueues the H2D copy of a device read (and
now the D2H copy of a staged write) on it, and the cuFile reactor records its
stream gate on it.  With cudaStreamPerThread both ordered the transfer after
the worker's own, empty, per-thread stream, so a read could land before work
the caller had queued ahead of it (and be overwritten by it), and a write
could read its source before the caller's producer ran.

templated_ioctx's dispatch now replaces every cudaStreamPerThread of a
request's device slices, on the submitting thread, by a non-blocking stand-in
stream (one per device and request) that waits there on an event recorded on
the caller's per-thread stream; the slice keeps the stand-in alive.  No
per-thread handle crosses threads any more.  The fs_cache's cache-hit device copies, enqueued on the caller's stream by the
submitting thread, were retired through a completion lane keyed by stream
handle; cudaStreamPerThread has one handle for every thread's stream, so one
thread's completion could retire another thread's copies (releasing pins and
resolving the read before its DMA finished).  A cache-hit read on the
per-thread stream is now retired by its own event, recorded on the submitting
thread after its copies.  The legacy default stream needs
nothing: it is one stream per device and synchronizes with every blocking
stream.

Regression tests hold the caller's per-thread (and, for coverage, legacy)
default stream on the host, queue a memset of the destination behind it and
check the device read's bytes, for the uring reactor and for both cuFile read
paths; a two-thread test holds one thread's per-thread stream under a cache-hit
read and checks another thread's cache-hit read neither retires nor waits for
it.
The io_uring reactor models io_write_reactor_c.  A write slice is split at
the write granule (max(4 KiB, page size)) into an unaligned head, an aligned
body and an unaligned tail: the body goes O_DIRECT straight from a congruent
host source or through the pinned staging blocks (a D2H copy on the caller's
stream for device sources, a memcpy for host ones), head and tail through
the page cache.  Writes use write / writev / write_fixed SQEs; short writes
are resubmitted, a zero-byte completion is EIO, and O_DIRECT or fixed-buffer
runtime errors fall back to buffered I/O as reads do.  Flush is an
IORING_OP_FSYNC with IORING_FSYNC_DATASYNC; the synchronous host_write and
flush run on the caller.  write_split_bytes (default 32 MiB) spreads large
slices over the selected reactors.

The worker never blocks on a caller's stream: pending D2H and H2D copies are
polled with a capped CQE wait, and staged-write bytes whose copy is pending
count in queued_bytes().  start() waits for the worker's ring setup and
rethrows its error (e.g. EPERM under seccomp); requests enqueued before or
after such a failure fail with that error, and start() may be retried.
Clean shutdown still cancels with operation_canceled.

Writable local objects grow atomically as writes complete and carry an
opened_handle identity.  The write open anchors its O_DIRECT descriptor to
the already-created buffered one through /proc/self/fd instead of reopening
the path, never repeats the destructive create / truncate, and fails with
descriptor_mismatch_error when a pathname fallback cannot prove both name the
same file; a test seam renames the path at that point.

Tests cover open modes, aligned / unaligned host and device scalar, vector
and mixed writes, holes and size growth, stream ordering, priorities,
concurrent block-sharing writers, shutdown and pre-start cancellation, the
mincore direct-write check, stalled streams, short writes, RLIMIT_FSIZE,
sibling errors, seccomp start-up failure, the anchored write open, and fs_cache
/ metadata invalidation by writes.
The cuFile reactor models io_write_reactor_c.  Write opens create / truncate
the buffered fd per write_mode, preallocate best-effort, and reopen the
O_DIRECT fd from it through /proc/self/fd (never by path; a fallback that
opens another file fails with descriptor_mismatch_error rather than repeat
the destructive open), then register it with cuFile under the usual compat
rules.  Writable objects grow atomically and carry an opened_handle identity.

Writes split at the write granule.  On a real GDS platform the aligned body
of a device source of at least gds_threshold goes to chunked synchronous
cuFileWrite after the slice passed its stream gate; head, tail, smaller bodies
and, on a compatibility-only platform, every device body are staged through
the double-buffered pinned scratch by D2H copies on the reactor's copy stream
and written with pwrite (O_DIRECT for aligned chunks).  Large host bodies
congruent with their offset go O_DIRECT from the caller; other host sources
go through the page cache with writeback started at once, and a host-only
vectored write goes to one reactor.  Short writes resubmit, zero-byte writes
are EIO, flush is fdatasync.  Staging copies drain before the scratch is
reused or a request settles, including when an event record fails after a
copy was queued.

cufile_api resolves cuFileWrite, cuFileBufRegister and cuFileBufDeregister.
config::register_min_bytes (default 0) opts into per-transfer buffer
registration through a process-wide, reference-counted table keyed by base
address.  Every non-empty cuFile read and write takes part in that table,
also with the knob at 0 (which only stops a context from creating
registrations), so a context that registers never covers memory another
context's unregistered transfer is using.  write_split_bytes defaults to 0 for cuFile.

Tests cover every GDS / staged / compat / unregistered / buffered write path,
writes ordered after the caller's per-thread and legacy default streams,
default and opt-in contexts sharing a buffer,
registration sharing and exclusion, caller stream ordering, progress of
unrelated host writes, shutdown cancellation, short and error paths and
8-byte-aligned host-block vectors.  The practices doc gains a writes section
(measured in compatibility mode only).
Pin null device streams to cudaStreamLegacy before dispatch, and gate cuFile on a recorded legacy-stream event instead of assuming cudaStreamQuery captures blocking-stream dependencies. Canonicalize cache-hit retirement lanes too.

Cover dispatch sentinels, null read/write streams and PTDS-built callers on uring and direct/staged cuFile paths. The deterministic enqueue seam fails without the fix; the normal legacy-built backend already passes the PTDS-caller test. An isolated PTDS-built cuFile backend reproduces early completion and stale writes without the fix (0ms, 0x11 written), and waits ~700ms with the fix (0xab written).
DISK memory spaces now read and write their files through a per-space
std::shared_ptr<io::ioctx> instead of the bespoke idisk_io_backend and its
pipeline worker.

- disk_memory_space_config gains a defaulted io_context, and
  reservation_manager_configurator::set_disk_mounting_point a defaulted
  fourth argument.  memory_space gets DISK-only, mutex-guarded
  get_io_context(), set_io_context() (non-null) and
  get_or_init_io_context(make_default); a failed or null factory installs
  nothing, and in-flight conversions keep their own reference.  The core only
  forward-declares ioctx: no core -> io link.
- cucascade/cudf/disk_io_context.hpp: make_disk_io_context() builds and
  starts an automatic / uring / cufile context (uring owns private portable
  pinned staging, 64 MiB per reactor), and default_disk_io_context() is the
  weakly cached, process-shared default adopted lazily on first conversion.
  automatic = cuFile with real GDS, else uring when this process may set up
  io_uring (also falling back when the reactors' own ring setup fails at
  start), else the cuFile reactor's POSIX path; forced uring fails with the
  setup errno.  The cuFile paths are gated on CUCASCADE_HAS_CUFILE.
- The four HOST/GPU <-> DISK converters open one object and issue one
  vectored request per table; the GPU read allocates the whole buffer tree
  before submission, waits once, and records the writer event.  Every
  allocation outlives its I/O on every unwind path; spill files keep their
  RAII removal and layout; no durability flush in the spill path.  The
  redundant post-read host stream sync is gone: a completed device read is
  host-observed.
- A cudf build now requires CUCASCADE_BUILD_IO (configure-time diagnostic);
  cudf-free and topology-only builds are unaffected.
- Removed: include/cucascade/data/{disk_io_backend,io_backend_registry}.hpp,
  src/data/{pipeline_io_backend.cpp,io_worker.hpp,io_backend_registry.cpp,
  io_backend_internal.hpp} and their tests.

AGENTS.md describes the io_context-based DISK tier.

BREAKING CHANGE: the custom-backend memory_space constructor and getter,
idisk_io_backend, io_batch_entry and the io backend registry are removed.
Three-field disk_memory_space_config aggregates and three-argument
set_disk_mounting_point calls still compile; pass an io::ioctx (or nothing,
for the shared default) instead of a backend.

Keep the source device current from wait_for_writer through the write: a null stream otherwise named two different default streams and the spill could read stale source bytes before its writer finished. Reject an explicit stream of another device with cucascade::logic_error before generating a file path.

Both regression cases ran on the 4xA100 host for uring and cuFile compatibility mode. Without this fix the null-stream test spills stale bytes on both backends; foreign streams yield a CUDA error on uring or no error on cuFile. With it, both cases pass (22 assertions).
- test_disk_randomized: randomized mixed / nested / empty tables with
  non-repeating data, round-tripped through HOST and GPU on the default
  context and on forced uring and cuFile contexts; only forced uring is
  skipped, and only where io_uring is proven unusable in this process.
- benchmark_disk_converter measures the DISK tier per io context with
  backend-neutral names (automatic, forced cufile, optional forced uring),
  CUCASCADE_BENCH_DISK_DIR, cold reads and an explicit benchmark-only
  fdatasync for comparable write numbers.  No throughput assertions.
…eries

The S3 write path needs PUT (object, UploadPart), POST
(CreateMultipartUpload, CompleteMultipartUpload) and DELETE
(AbortMultipartUpload) requests, most with a sub-resource query
(?uploads, ?partNumber=N&uploadId=U, ?uploadId=U).

- request_method gains PUT, POST and DELETE, plus to_string();
- request_spec describes one object-level request (method, pre-encoded
  canonical sub-resource query, whether it has a body);
- request_authorizer::authorize_request(spec, timeout): a new virtual whose
  default delegates plain GET / HEAD to authorize() (existing authorizers
  keep working) and throws credential_error for anything else;
- the SigV4 header and presigned authorizers implement it: the query is
  merged into the canonical request, a body is signed as UNSIGNED-PAYLOAD
  (sha256("") without one), and X-Amz-* keys in the query are rejected
  (no signing-parameter smuggling).

Tests ([s3][sigv4]): canonical requests and signatures for every write
request kind against independently computed SigV4, query validation, and
the mock authorizer's recording of request specs.
S3 writes without a transport (src/io/rest/s3/multipart): nothing here
performs I/O, allocates pinned memory or touches CUDA.

- protocol helpers: the S3 limits (10000 parts, 5 MiB minimum part, 5 GiB
  part / single PUT), part numbering for a fixed part size, the canonical
  sub-resource queries of CreateMultipartUpload / UploadPart /
  CompleteMultipartUpload / AbortMultipartUpload, the Complete request
  XML, and parsers for InitiateMultipartUploadResult,
  CompleteMultipartUploadResult (including a 200 OK carrying an <Error>
  body, which S3 says to retry) and part ETags;
- upload_state: the thread-safe state machine of one object being written
  (parts assembled from out-of-order writes, uploaded, retried, committed
  as one PUT for an object of at most one part or as a multipart upload
  otherwise, or aborted), which the REST reactor drives.

This includes NVIDIA#221's fix 5ff07f6: a Complete 2xx without a result
document fails.  interpret_complete_response used to accept any 2xx body
that was not an S3 <Error> as success, so a proxy page or keep-alive
whitespace yielded an empty ETag and the object counted as committed
although the commit was never acknowledged.  A
CompleteMultipartUploadResult element is now required (attributes such
as xmlns accepted); anything else is http_bad_message and not retried.
A result without an ETag is still accepted, for S3-compatible stores
that omit it.

Tests ([s3]): part planning, request and response XML ("multipart:
CompleteMultipartUpload response (200 OK + <Error>)"), ETag handling, and
upload_state transitions under concurrent writers, failures and aborts.
…s in the transport

What the S3 write path needs from the io_uring transport before it sends
PUT and POST bodies:

- The stall detector counts request body bytes sent as well as response
  bytes received.  It used to count received bytes only, so an upload
  slower than one stall window was cancelled while it was still sending
  (curl's low-speed check measures the transfer in either direction).  A
  server that never answers still stalls.
- exchange_result::request_sent: the head and the whole body were written.
  With a transport error and no response byte the server may have acted
  on a request whose answer was lost; CreateMultipartUpload must never be
  re-sent then, and a conditional commit that retries into a 412 cannot
  tell whether its own earlier attempt took effect.
- exchange_result::error_body_complete and response_meta::amz_error_code:
  a drained error body's S3 <Code> (http::extract_error_code), and whether
  the body was drained to its end.  A 404 alone does not say NoSuchUpload
  from NoSuchKey, and a cut body must not be read as a known code.
  Accepted (2xx) bodies stay the caller's to interpret.
- exchange::continue_timeout: with Expect: 100-continue, the held body goes
  out anyway when no interim or final response arrived in time (a server
  that ignores Expect; curl's EXPECT_100_TIMEOUT).  The wait is a POLL_ADD
  that the deadline cancels softly (both CQEs reaped before the body is
  sent), so a user-space TLS stream's receive state is never disturbed.
- The user-space TLS stream seals stream_options::tx_batch_records (16)
  records per ciphertext SEND instead of a fixed 3: one SEND is in flight
  at a time (record order), so a 16 MiB part took ~340 SEND round trips;
  now ~64.  The FIFO holds 17 records; it is not value-initialised, so a
  connection that never sends a body touches one page of it.

Tests ([rest][net]): a slowly read 8 MiB upload under a 1 s stall window
completes (and fails without the send-side progress), a silent server
still stalls, request_sent on a refused connection and on a dropped
response, the error code of a drained 404 and an incomplete cut body, a
held body sent after continue_timeout with the connection reused after,
and the sealer's record budget per stream_options.
The write half of NVIDIA#221's loopback S3 fixture, re-applied onto the read
server of the io_uring reactor (HTTPS mode, mid-body KeyUpdate and drop
policies, chunked GET bodies, trailing garbage, data-GET waits are kept):

- an in-memory object store: PutObject (If-None-Match: * -> 412 when the
  key exists), CreateMultipartUpload, UploadPart, CompleteMultipartUpload
  (part order, ETags and the minimum part size validated),
  AbortMultipartUpload and DeleteObject; written keys are what later
  GET / HEAD / LIST see, missing_key_404 turns other keys into NoSuchKey;
- write faults (write_kind, write_fault: status, ok_with_error,
  ok_with_body, drop_mid_body, drop_after_body, hold), Expect:
  100-continue handling (honor / ignore / reject), keep-alive whitespace
  before the Create / Complete XML;
- counters (put_object_count, create_upload_count, upload_part_count,
  complete_upload_count, abort_upload_count, delete_object_count,
  active_upload_count, ...), request records and put_stored_object
  seeding.

ETags are S3's: the MD5 of a body for a single PUT and a part (hashed
outside the store lock, so uploads of several connections hash in
parallel), md5(concat(binary part MD5s))-N for a completed upload, so a
client can verify a commit against its part ETags.

Every public name is NVIDIA#221's (the REMOTE tier's test utilities use them),
s3_error_body is public, and a constructor overload keeps the read-only
HTTPS call sites (object, faults, listing, scripts, tls) unchanged.  No
library change.

Tests ([s3]): test_loopback_s3_server.cpp, the fixture itself, against
independently signed requests and raw sockets.
…art upload, commit on flush

Enable writes on the REST backend, replacing the not-implemented stubs:
rest_reactor models io_write_reactor_c with supports_device_write (the
reactor_write_traits specialization is gone), so rest_ioctx reports
supports_write() and supports_device_write().  The upload state machine
(s3::upload_state) decides; the reactor's worker moves bytes and settles
every claim exactly once.  The decision logic is NVIDIA#221's one-for-one; the
plumbing is rebuilt on the slot / exchange model of the read reactor.

- Open: create_or_truncate and create_new (If-None-Match: * on the final
  PUT / Complete); create_or_open and open_existing are not_supported.  No
  request at open; size_hint raises the part size past 10000 parts.
- Writes land in parts of write_split_bytes.  A host slice covering a part
  is uploaded from the caller's buffer; anything else is copied (memcpy on
  the worker, or D2H on the caller's stream, polled like the H2D copies of
  reads) into a part buffer from the context's host resource; the write
  completing a part uploads it.  rest_ioctx cuts slices at part boundaries.
- Part buffers count as staging: on exhaustion a write parks while write
  work or staged reads of the reactor are in flight and retries when they
  settle or a buffer is released (on any thread: the release wakes the
  loop); with nothing in flight it fails with rmm::out_of_memory.  A
  parked staged read no longer ends the submit pass (it used to `break`):
  the write calls behind it, which return the blocks, still get their
  connections, and wait() never blocks while a write call could start;
  otherwise a burst of parts and a staged read sharing a small pool
  deadlocked.  Write and flush requests have their own intake, expanded
  while reads are parked (most writes need no staging).
- flush / flush_async commit (one flush slice on the least-backlogged
  reactor; rest_ioctx::flush_io waits on flush_async_io): the remaining
  parts (gaps zeroed; never-written parts from a static 1 MiB zero block in
  windows of 64 iovecs, so a zero body can no longer be freed while it is
  sent: NVIDIA#221's b9d9090), then CompleteMultipartUpload, or one PUT for an
  object of at most one part.  The path's cached bytes and metadata are
  invalidated once the commit settled (ioctx::invalidate_cached is
  protected for that).
- Each S3 request is an s3_call with its own retry envelope on the
  worker's connections.  A call is owned by the worker's queues or slot and
  owns its job (a part, a flush, a Create), never the reverse (NVIDIA#221's
  a16aa5a): the last call of a failed commit releases the job, and with it
  the session whose destruction aborts the upload.  Work queued during a
  pass, including a Create's follow-up UploadParts and the waiters its
  settlement hands back, starts in that pass (NVIDIA#221's fc28071 invariant).
- Rules (owner's decisions): PUT / UploadPart re-sign a 403 up to
  max_auth_retry_attempts; Create / Complete / Abort stay final on 403.
  CreateMultipartUpload is never re-sent once it went out whole and its
  answer was lost, except for the free stale retry (a pooled connection
  that died before any response byte: the request was not read).  The
  object's session remembers, across flushes, that a commit request lost
  its answer: a create_new commit answering 412 then reports io_error
  ("commit outcome unknown"), never EEXIST, and a Complete answered
  NoSuchUpload is checked with one HEAD against md5(concat(binary part
  MD5s))-N of the recorded part ETags; a mismatch (or non-MD5 part ETags)
  fails the flush as "outcome unknown".  Without a lost answer a
  NoSuchUpload fails the flush.  A 409 ConditionalRequestConflict on a
  conditional commit is retried.  A 2xx Complete without a result
  document fails (NVIDIA#221's 5ff07f6).  Never a false commit success.
- Cross-reactor work: parts of one object may run on several reactors;
  work parked on the object and released by a settlement runs on the
  settling reactor in that pass when it parked there, and is handed back
  to the reactor that parked it otherwise (an intake queue and an eventfd
  wake; it starts in the pass that wakes it), so the parts of a burst
  dispatched before the UploadId exists upload on every reactor instead
  of all on the one that created it.  The abort of an object destroyed
  uncommitted goes through an abort sink owned by rest_ioctx that posts a
  control job (signed from a request spec, so DeleteObject needs only a
  new kind) to a running reactor.  Lifetime: the sink's close (first in
  ~rest_ioctx) makes every write object still alive give back its part
  buffers while the host memory resource lives; such an object then fails
  its writes and flushes with operation_canceled.
- Config: write_split_bytes (16 MiB), expect_continue (off),
  continue_timeout (1 s), complete_timeout_s (300), max_write_connections
  (0 = all).  Stats: puts, put_bytes, upload_parts, upload_part_bytes,
  zero_part_bytes, creates, completes, complete_error_retries, aborts,
  commits, commit_failures, write_failures, d2h_copies, part_buffer_waits,
  part_buffers_live.  Test hooks: on_call_started / on_call_finished.
- The reactor_worker class moves to src/io/rest/reactor_worker.hpp (no
  behaviour change) so the write path is its own TU.

Tests ([rest][write]): every NVIDIA#221 case, including the regressions "aborts
an upload whose commit failed once the object is destroyed" (a16aa5a), "a
200 Complete without a result document fails the flush; flushing again
commits" (5ff07f6) and "starts a write's first request without waiting
for a loop tick" (fc28071, now asserted with the worker's wait count);
"uploads parts parked on another reactor's CreateMultipartUpload on their
own reactor" replaces the mailbox case.  New: the zero-body generator
and a 72 MiB never-written part, the multipart ETag rule, 403 re-sign on
UploadPart and final on Create, a lost conditional PUT then 412, a lost
Complete recovered by HEAD and a NoSuchUpload that is not ours, a PUT torn
mid-body retried on a new connection, part-buffer parking and the
out_of_memory rule, device writes on every GPU, HTTPS writes on the
user-space record layer (kTLS required skips on hosts without it),
max_write_connections, expect_continue (honored and ignored), write
stats, UploadParts starting in the Create's pass while a staged read is
parked on a shared small pool (deadlocked before), a lost conditional
answer of an earlier flush then 412, a lost Complete then a failed flush
then NoSuchUpload, a Create on a stale pooled connection, writes and a
flush beside a parked read, an ioctx destroyed before a half-written
object, and a hidden throughput benchmark ([bench_rest_write]).
test_rest_write_unimplemented.cpp is gone with the stubs.
…urce device

Bugs of the converters that copy a GPU table's buffers one by one
(GPU -> host_data_representation, GPU -> DISK and GPU -> GPU); they predate
the io stack split (plan_column_copy and the fast GPU -> host converter come
from NVIDIA#52, the GPU -> GPU column walk from NVIDIA#52 / NVIDIA#116, the GPU -> DISK
converter from NVIDIA#96, view-backed gpu_table_representation from NVIDIA#116).

- Sliced sources. A view-backed representation may hold a slice, but the
  serializers copy every buffer of a column and its children from the start,
  and plan_column_copy() / reconstruct_column_p2p() only asserted offset 0 in
  debug builds: in release builds a nullable slice starting at row 37
  round-tripped as the table's leading rows with unshifted validity, and a
  prefix slice (offset 0) kept the extent of the table it was cut from in its
  children (DICTIONARY32 indices, STRING / LIST offsets, STRUCT fields), so it
  was restored with the wrong row count (GPU -> GPU failed with
  cudaErrorInvalidValue or "Invalid offsets column size" instead, or, for a
  fixed-width slice, silently returned the wrong rows). needs_compaction()
  detects both (non-zero offset anywhere, or a child extent that does not
  match its parent's rows), and serializable_view() compacts such a table
  first (cudf::table's deep copy), with the source device current, after the
  writer ordering; the copy lives until the transfer completed. GPU -> host
  and GPU -> DISK compact on the conversion stream; GPU -> GPU compacts on a
  stream of the source space and synchronizes it before the cross-device
  copies on the target stream read the copy. It needs device memory for the
  whole table on the source GPU; if that fails, the conversion fails and the
  batch stays where it is.
- Failures after enqueued work. If the compaction (or anything later) throws
  after work reading the source was enqueued on the conversion stream, the
  stream is synchronized (drain_on_unwind, under the source-device guard)
  before the exception propagates, so the caller may release or mutate the
  source. GPU -> GPU does the same for its target stream: a failed allocation
  for column i+1 no longer returns while column i's peer copy is still queued
  (also before the compacted copy is freed), and the host-staged peer copy
  drains its stream before its pinned blocks go back to the pool.
- Source device. GPU -> host_data made the source device current only around
  the compaction; it now keeps it current from the writer ordering through
  the copies and the release of the compacted copy. GPU -> DISK already
  does (and refuses a stream of another device); the compaction now runs in
  that scope too.

Regressions [disk]: view-backed nested nullable slice [37, 2011) and prefix
slices [0, 10) of a 100-row DICTIONARY32 / STRING / LIST / STRUCT table
through host_data and DISK; a compaction whose second column allocation is
refused (seam: a memory resource refusing large allocations) leaves the
conversion stream idle when the error arrives, on both routes; [multi_gpu]: a
GPU 1 table (also sliced) written late by a non-blocking stream spilled to DISK
from GPU 0 on the default stream, a GPU 1 table converted to host_data from
GPU 0 on a GPU 1 stream, nested / prefix / fixed-width slices converted GPU 0
-> GPU 1 and GPU 1 -> GPU 0 and back, and failed GPU -> GPU conversions (an
allocation refused on the target, a compaction refused on the source) that
leave the source's writer finished when the error arrives. The test sources
that own a raw stream move it out of the moved-from object.
Add ioctx::remove_io_object_async / remove_io_object: the front end strips a
file: scheme, invalidates the path in the ioctx's cache and metadata store
before the request and again once it settled (as the write front ends do),
and forwards to the new protected hook remove_io_object_async_io. The default
hook resolves with not_implemented_error (errc::not_supported): the local
backends do not implement it (DISK spaces keep removing their files with
std::filesystem).

rest_ioctx implements it with S3 DeleteObject on its control reactor (least
backlogged, ties rotated): rest_reactor::delete_object_async posts a
control_job of the new kind delete_object (DELETE /bucket/key, signed from its
request spec through authorize_request, like AbortMultipartUpload), retried
like the other control jobs with max_retry_attempts. Classification: 2xx is
success; a 404 NoSuchKey, or a 404 without an error code whose error body was
read to its end, is success too (the key is gone, also when the 404 arrives
while the reactor stops); any other 404 (NoSuchBucket, ...) fails; a 404 whose
error body was cut or not drained (the code cannot be known) is retried like a
transport error (finish_control); a 403 is final. io_priority::low marks the
job background: it starts only when no reads or writes wait, behind data work
instead of ahead of it like blocking control calls (the reactor has no
separate bulk class).

The job owns the promise its on_done settles, and the intake / slot / retry
queue own the job, so the returned future may be dropped without any
reactor-owned bookkeeping. Before start() / after shutdown() (or from a
reactor thread) it runs blocking on a private helper like head_object and
returns a ready future. A delete queued or in flight at shutdown fails with
operation_canceled (intake::close / drain). Stats: deletes (attempts sent).

Tests: test/io/rest/test_rest_delete.cpp [rest][delete] -- 204 (presigned
and header signing), 404 NoSuchKey and code-less 404 ok, 404 NoSuchBucket
fails without a retry, a 404 whose (oversized) error body was not read is
retried, 403 final, 503-then-204 retry, exhausted retries, unstarted reactor,
dropped futures (incl. many low-priority ones) settle, shutdown with deletes
in flight, non-s3 path, uring -> not_supported. The loopback server's status
fault takes an optional body for the oversized 404.
Tier gains REMOTE, appended before SIZE (GPU 0, HOST 1, DISK 2 unchanged,
REMOTE 3, SIZE 3 -> 4). A REMOTE space keeps spilled batches as objects
under an object-store prefix; the core library stays io-free (it only
stores the io context pointer, as for DISK).

- memory/config.hpp: remote_memory_space_config {remote_id, memory_capacity
  (default unbounded_remote_capacity = 1 PiB), uri_prefix, io_context},
  thresholds as DISK (trigger = capacity, stop = 0.99 x capacity); appended to
  the memory_space_config variant (existing indices unchanged); public
  validate_remote_uri_prefix() (s3://bucket[/prefix]; rejects other schemes,
  empty bucket, userinfo / port in the bucket, query / fragment, whitespace and
  control characters, empty segments; returns it normalized).
  ABI note: memory_space_config gains an alternative. It is appended, so the
  existing alternatives keep their indices, but the variant's size and layout
  change: code compiled against the old header must be rebuilt.
- memory_space: REMOTE constructor (reuses disk_access_limiter: capacity
  bounds the bytes of reservations in flight, not the bytes stored), new
  get_remote_uri_prefix(); get/set/get_or_init_io_context accept DISK or
  REMOTE; to_string prints REMOTE.
- reservation_manager_configurator::add_remote_space(remote_id, uri_prefix,
  capacity = unbounded_remote_capacity, io_context = nullptr); build() emits
  REMOTE configs after the DISK ones.
- memory/disk_table.hpp: generate_remote_object_uri(prefix) ->
  {prefix}/{hostname-pid-start_ms}/batch_{seq:016x}_{rand:08x}.cucascade (no
  I/O), implemented in src/memory/remote_uri.cpp.

Tier audit (switches have `default:`, so the compiler does not flag missing
REMOTE cases; every use in the tree checked by hand):
  changed here:
  - memory/common.hpp Tier enum
  - memory/config.hpp *_config::tier(), memory_space_config variant
  - memory_space.cpp ctor overloads, get/set/get_or_init_io_context (DISK|REMOTE),
    to_string switch (+REMOTE); get_disk_mount_path stays DISK-only
  - reservation_manager_configurator build() (REMOTE after DISK)
  - test/utils/mock_test_utils.hpp make_mock_memory_space (+REMOTE)
  changed in the next commit (cudf layer):
  - cudf/bandwidth_profiler.cpp canonical_type_for switch (default throws),
    should_skip_pair DISK rules; test_bandwidth_profiler.cpp operator<< switch
  - cudf/representation_converter_builtins.cpp registry (REMOTE converters)
  no change needed (GPU/HOST-specific or tier-generic):
  - memory_space.cpp dtor (HOST pool unregistration), make_reservation* /
    get_available_memory / ... visits over the reservation allocator variant
  - memory_reservation.hpp tier_memory_resource_trait (void, like DISK)
  - memory_reservation_manager.{hpp,cpp} config visit and Tier-keyed lookups,
    request strategies (any_memory_space_in_tier[s], *_to_downgrade/upgrade)
  - memory/common.cpp make_default_allocator_for_tier (null resource branch)
  - memory/topology_index.hpp GPU/HOST id extraction
  - memory/fixed_size_host_memory_resource.cpp HOST space id
  - data/data_batch.hpp GPU-only checks (:398, :677), data/common.hpp
  - io/cache/types.cpp, io/datasource_factory.cpp HOST pool lookups
  - cudf/host_data_representation.cpp HOST resource, bandwidth_profiler GPU
    stream-owner checks, cudf/bandwidth_profiler.hpp GPU defaults
  - benchmark/benchmark_{disk,representation}_converter.cpp (DISK/GPU/HOST only)
  - tests enumerating explicit tier subsets (test_memory_reservation_manager,
    test_data_*), test_disk_memory_space accessor rejection (GPU)
  docs (AGENTS.md, README.md: "three tiers") are updated with the docs commit.

Tests: test/memory/test_remote_memory_space.cpp [memory][remote].
… context

HOST and GPU batches can now be converted to and from REMOTE memory spaces:
each batch becomes one object in an S3(-compatible) store.

- cucascade::remote_data_representation (core, io-free): the column layout
  (disk_table_allocation, file_path = object URI) and an opaque handle to the
  written object (detail::remote_object, defined by the cudf layer). clone() is
  unsupported (as DISK).
- Object removal: releasing the handle never does I/O or blocks (it may run on
  a reactor thread of its own context); it hands the context, the committed
  write object and the URI to a process-wide removal queue. Its thread (never
  a reactor thread) issues the DeleteObject (io_priority::low), owns the
  context reference until the removal settled and releases it there: removals
  survive the spaces / managers / representations that held their context (a
  context's destruction would cancel them), and a context's last reference is
  never dropped on its own reactor thread. The queue is created eagerly (by
  make_remote_io_context and before every REMOTE write), entries are allocated
  before the context is moved in (a failed allocation keeps the reference for
  good rather than releasing it on the caller's thread) and moved by splicing,
  and its thread never terminates on an exception. Failures are WARNed
  (rate-limited) and left to the bucket's lifecycle rule. Each pass first
  releases settled removals, then issues at most the ones queued when it
  began, one at a time. Public drain_remote_object_removals() waits for all of
  them (also run at exit; limits documented: late static releases, no
  deadline, unbounded, no fork). The scoped, bounded
  drain_remote_object_removals(contexts, timeout) is what embedders tear down
  with: it waits only for the removals of the given contexts (counted per
  context), for at most the timeout; at the deadline, removals still queued
  for them are never issued, each context is shut down (settling its calls in
  flight) and their entries are released on the calling thread, independent
  of the queue's thread; it reports how many were abandoned (WARN; left to the
  lifecycle rule) and whether it timed out.
- make_remote_io_context(remote_io_context_options) (cudf/remote_io_context.hpp):
  a started REST context (no fs_cache, no default context) owning its pinned
  staging (n_reactors x staging_bytes_per_reactor, 1 MiB blocks; the
  base-from-member pattern of make_disk_io_context). Validates the endpoint
  (exactly http(s)://host[:port]), region and keys (std::invalid_argument),
  that the staging holds two parts, and that io_uring is usable
  (std::system_error). Applies object_store CA/verify over rest.tls as the
  registry does.
- io::rest::s3::make_authorizer(object_store_config) is now public and shared
  by the io_context registry and make_remote_io_context.
- Converters HOST<->REMOTE and GPU<->REMOTE, factored with the DISK ones over
  layout_{host,gpu}_batch / restore_{host,gpu}_batch (same 4 KiB-aligned
  layout, metadata in memory). Writing (src/cudf/remote_io.cpp): a fresh name
  (generate_remote_object_uri), open create_new, the slices plus zeros for the
  alignment gaps so every part completes while the batch is written, flush =
  commit (single PUT or CompleteMultipartUpload). Contexts from
  make_remote_io_context write in windows of whole parts that fit half their
  staging and admit concurrent conversions against it (exec::admission_control),
  so a batch larger than the staging or concurrent spills never exhaust it.
  Device reads are counted in staging blocks (every GET allocates whole
  blocks; the GET count comes from the new public io::rest::max_read_gets(),
  the reactor's segment planner's worst case: its smallest target,
  max(1, n / min_segment_bytes, ceil(n / max_segment_bytes)), sharing the
  count with physical_ranges; GETs are not cut at part boundaries),
  so a wide table of tiny buffers is read in windows whose block count fits
  half the staging, each admitted for exactly that. A conditional commit
  refused with EEXIST, or one whose outcome the store left unknown (the new
  public io::rest::may_be_commit_outcome_unknown: an attempt's answer was lost
  and the retry got 412, or a lost Complete could not be verified; reported
  as io_error, never EEXIST), renames once and retries; a key left by an
  unknown outcome is not deleted (it could be another writer's) and is left
  to the bucket's lifecycle rule. Any failure destroys the unflushed object
  (the upload is aborted) and rethrows, leaving the source intact. Reads go
  through the committed write object (known size, If-Match: no HEAD), with
  bytes-delivered checks; a failed restore leaves the REMOTE representation
  valid. A REMOTE space without an io context -> std::logic_error. GPU
  sources: the source device is current from the writer ordering through the
  write (a null stream resolves to that device's default stream for both; an
  explicit stream of another device is refused, as for DISK), a
  sliced view-backed source is compacted first (layout_gpu_batch, shared with
  GPU -> DISK), and a failure after work reading the source was enqueued
  drains the conversion stream before it propagates.
- bandwidth profiler: canonical type REMOTE -> remote_data_representation;
  storage tiers (DISK, REMOTE) are never paired with each other, and REMOTE
  pairs are measured only with the new bandwidth_profile_config::
  measure_remote_pairs (default false).

Tests ([remote]): test/data/test_remote_converters.cpp (loopback S3 server:
round trips GPU and HOST of numeric, strings + nulls, lists/structs with nulls,
nullable, sliced, empty, single PUT, multi-part, larger than the staging, one
HTTPS case, view-backed sliced nested nullable source, GPU 1 source with a
late non-blocking writer converted from GPU 0 on the default stream
[multi_gpu] (also sliced), an explicit stream of another GPU refused before
any upload [multi_gpu], view-backed prefix slices (dictionary / string /
list / struct), a compaction failing part-way (stream drained), wide table of
tiny columns with slow GETs and 4 KiB segments (staging windows);
DeleteObject on destruction, removal outliving its space and manager
(throttled retry after the manager's teardown), release on a reactor thread of
the representation's own context, delete failure only warns; the scoped
drain: cancels a held removal at its deadline and releases the context, waits
only for its own contexts, never issues a removal still queued and does not
depend on the queue's thread (blocked in a removal through a shut-down
context), cancels nothing when settled in time, returns at once with a zero
timeout and nothing outstanding; persistent UploadPart 5xx and Complete 5xx -> throw +
abort + no object; 412 collision rename (PUT and Complete) and twice -> fail;
unknown commit outcome (lost PUT answer then 412; Complete NoSuchUpload not
verified) -> one retry under a new name;
GET failure on restore -> throw, representation intact; no io context ->
logic_error; concurrent spills over shared staging),
test/data/test_remote_io_context.cpp, and a REMOTE case in
test_bandwidth_profiler.cpp; shared helpers in test/utils/remote_test_utils.hpp;
max_read_gets against the reactor's GET count in test_rest_read_robustness.cpp
([rest]).
- docs/io/remote-tier-design.md: configuration (space prefix vs. io context,
  no default context, endpoint/credential validation), the memory_space_config
  variant change (appended alternative: indices kept, size/ABI changed),
  capacity semantics (in-flight bytes, no stored-bytes quota), object naming
  and layout, spill and restore steps (source device current from the writer
  ordering through the write, sliced sources compacted), staging windows
  (writes in whole parts, device reads counted in staging blocks) and
  admission, failure handling, lifecycle (the process-wide removal queue that
  owns contexts until their deletes settled, drain_remote_object_removals()
  and the scoped, bounded drain_remote_object_removals(contexts, timeout)
  embedders tear down with)
  and the requirement of an S3 lifecycle rule (expiration +
  AbortIncompleteMultipartUpload) for objects left by crashes or failed
  deletes; what is not implemented.
- docs/io/rest-transport.md "Writes": DeleteObject (a control job on the
  reactor; 2xx, 404 NoSuchKey and a complete code-less 404 are success, other
  404s fail, a cut 404 is retried; io_priority::low behind data work) and the
  `deletes` counter. remote-tier-design.md describes the segment planner's GET
  bound and the fresh-key retry after an unknown commit outcome.
- README, docs/README, ARCHITECTURE.md (tier table and diagram), AGENTS.md,
  memory-management.md, bandwidth-profiler.md: the fourth tier.
giancao added a commit to giancao/cuCascade that referenced this pull request Nov 14, 2026
Mark the places where NVIDIA#221 (S3 writes), NVIDIA#222 (object deletion, REMOTE
tier) and NVIDIA#224 (NUMA reactor placement) would land, so the gaps are
visible in the code.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants