Skip to content

fix(exec,io): threads started without CPUs run on the process's CPUs - #243

Draft
aocsa wants to merge 1 commit into
NVIDIA:mainfrom
aocsa:fix/exec-process-affinity
Draft

aocsa wants to merge 1 commit into
NVIDIA:mainfrom
aocsa:fix/exec-process-affinity

Conversation

@aocsa

@aocsa aocsa commented Nov 6, 2026

Copy link
Copy Markdown

Description

A thread inherits the CPU affinity mask of the thread that starts it. On hosts with more than 64 cores, DuckDB pins each of its worker threads to one core (pin_threads = auto). Sirius starts cuCascade's io_uring and REST reactors, and the fs_cache completion pool, from whichever thread runs the first statement. When that is a pinned DuckDB worker, the reactor thread, every io_uring worker it creates, and the pool's threads all share that one core.

On a 144-core GB200 host, two of four Sirius processes hit this. They read GPFS 20-25% slower than the other two and set the query time on every scan-bound TPC-H query. With the fix, TPC-H Q5 at SF10000 on 4x GB200 went from 43.5-44.8 s to 34.6-36.5 s warm.

What changes:

  • thread_util::get_process_affinity() returns the main thread's mask, whichever thread calls it. It is not noexcept, unlike its neighbours: static_thread_pool's constructor calls it, and an allocation failure there has to propagate (the existing allocation-failure sweep test covers it).
  • static_thread_pool uses that mask when given no CPUs.
  • The uring and REST reactor threads set it on themselves before their first submission, so io_uring's workers inherit it.

Nothing changes on hosts where the starting thread is not pinned: the process mask is what those threads inherited anyway.

The Sirius side (its own thread pools, and the submodule bump) is sirius-db/sirius#2485.

Overlap: the io/reactor-cpu-isolation branch rewrites rest_reactor::start(); this PR's change there is three lines and rebases trivially.

Testing

  • New [exec][thread_pool] test: a static_thread_pool built from a thread pinned to one CPU runs its worker on the process's CPUs.
  • New [io][uring][affinity] test: a uring reactor started from a thread pinned to one CPU runs its read completions on the process's CPUs.
  • Both fail without the fix (worker on CPU 0 only) and pass with it.
  • The REST reactor change is the same two lines as the uring reactor's and has no dedicated test.
  • Full cucascade_io_tests (cudf off) on an RTX PRO 4500 box: everything passes except two test_datasource_registry cases that expect the kvikio backend. They fail the same way on main a11ff66 in a cudf-off build.

A thread inherits the affinity mask of the thread that starts it. DuckDB
pins each worker thread to one core on hosts with more than 64 cores
(pin_threads = auto), and Sirius starts its io_uring and REST reactors
and the fs_cache completion pool from whichever thread runs the first
statement. On a 144-core GB200 host, a process that started them on a
pinned DuckDB worker ran its reactors and every io_uring worker on that
one core, and read GPFS 20-25% slower than its peers.

- thread_util::get_process_affinity() reads the main thread's mask.
- static_thread_pool uses it when given no CPUs.
- The uring and REST reactor threads take it before their first
  submission, so io_uring's workers inherit it.

Test: a static_thread_pool and a uring reactor started from a thread
pinned to one CPU run their workers on the process's CPUs.
@copy-pr-bot

copy-pr-bot Bot commented Nov 6, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

ivanmilevtues added a commit to CodeBoarding/cuCascade that referenced this pull request Nov 9, 2026
Use the process main thread's CPU mask for default worker placement. Apply names and affinity before initialization and reactor work, propagate pool initializer failures, and join workers on partial startup failure. Prepare reactor names and masks before spawning so allocation failures remain recoverable.

Inspired by NVIDIA/cuCascade PRs NVIDIA#240 and NVIDIA#243; retain the existing pool API and cover startup ordering, failure cleanup, and real REST worker affinity.

NVIDIA#240
NVIDIA#243

Amp-Thread-ID: https://ampcode.com/threads/T-01a1c140-9c31-7237-bc6b-617fa475f82a
Co-authored-by: Amp <amp@ampcode.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant