Repository navigation
Conversation
A thread inherits the affinity mask of the thread that starts it. DuckDB pins each worker thread to one core on hosts with more than 64 cores (pin_threads = auto), and Sirius starts its io_uring and REST reactors and the fs_cache completion pool from whichever thread runs the first statement. On a 144-core GB200 host, a process that started them on a pinned DuckDB worker ran its reactors and every io_uring worker on that one core, and read GPFS 20-25% slower than its peers. - thread_util::get_process_affinity() reads the main thread's mask. - static_thread_pool uses it when given no CPUs. - The uring and REST reactor threads take it before their first submission, so io_uring's workers inherit it. Test: a static_thread_pool and a uring reactor started from a thread pinned to one CPU run their workers on the process's CPUs.
3 of 4 tasks
ivanmilevtues
added a commit
to CodeBoarding/cuCascade
that referenced
this pull request
Nov 9, 2026
Use the process main thread's CPU mask for default worker placement. Apply names and affinity before initialization and reactor work, propagate pool initializer failures, and join workers on partial startup failure. Prepare reactor names and masks before spawning so allocation failures remain recoverable. Inspired by NVIDIA/cuCascade PRs NVIDIA#240 and NVIDIA#243; retain the existing pool API and cover startup ordering, failure cleanup, and real REST worker affinity. NVIDIA#240 NVIDIA#243 Amp-Thread-ID: https://ampcode.com/threads/T-01a1c140-9c31-7237-bc6b-617fa475f82a Co-authored-by: Amp <amp@ampcode.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
A thread inherits the CPU affinity mask of the thread that starts it. On hosts with more than 64 cores, DuckDB pins each of its worker threads to one core (
pin_threads = auto). Sirius starts cuCascade's io_uring and REST reactors, and thefs_cachecompletion pool, from whichever thread runs the first statement. When that is a pinned DuckDB worker, the reactor thread, every io_uring worker it creates, and the pool's threads all share that one core.On a 144-core GB200 host, two of four Sirius processes hit this. They read GPFS 20-25% slower than the other two and set the query time on every scan-bound TPC-H query. With the fix, TPC-H Q5 at SF10000 on 4x GB200 went from 43.5-44.8 s to 34.6-36.5 s warm.
What changes:
thread_util::get_process_affinity()returns the main thread's mask, whichever thread calls it. It is notnoexcept, unlike its neighbours:static_thread_pool's constructor calls it, and an allocation failure there has to propagate (the existing allocation-failure sweep test covers it).static_thread_pooluses that mask when given no CPUs.Nothing changes on hosts where the starting thread is not pinned: the process mask is what those threads inherited anyway.
The Sirius side (its own thread pools, and the submodule bump) is sirius-db/sirius#2485.
Overlap: the
io/reactor-cpu-isolationbranch rewritesrest_reactor::start(); this PR's change there is three lines and rebases trivially.Testing
[exec][thread_pool]test: astatic_thread_poolbuilt from a thread pinned to one CPU runs its worker on the process's CPUs.[io][uring][affinity]test: a uring reactor started from a thread pinned to one CPU runs its read completions on the process's CPUs.cucascade_io_tests(cudf off) on an RTX PRO 4500 box: everything passes except twotest_datasource_registrycases that expect the kvikio backend. They fail the same way onmaina11ff66in a cudf-off build.