Repository navigation
Description
CPU backend crashes with STATUS_ILLEGAL_INSTRUCTION on any CPU placement (regression: master-656 → master-660)
Summary
On the master-660-d2797b8 Windows CUDA build, any use of the CPU backend
— whether via the legacy --offload-to-cpu flag or the newer
--backend / --params-backend module-placement syntax — crashes the process
with STATUS_ILLEGAL_INSTRUCTION (0xC000001D, exit code 132 under bash /
3221225501 unsigned).
The crash happens after both backends initialize successfully, the moment a
tensor is actually placed on / executed through the CPU backend. The pure-CUDA
path (no CPU placement) works perfectly.
This is a regression: the exact same command works on master-656-0e4ee04
and crashes on master-660-d2797b8.
Environment
- OS: Microsoft Windows 10 Pro, version 10.0.19045 (build 19045), x64
- CPU: 12th Gen Intel Core i7-12800HX (Alder Lake-HX), 16 cores / 24 threads
- CPU feature flags (from sd-cli's own startup log):
SSE3=1 | AVX=1 | AVX2=1 | AVX512=0 | AVX512_VBMI=0 | AVX512_VNNI=0 | FMA=1 | F16C=1
— note: no AVX-512 - GPU: NVIDIA GeForce RTX 3080 Ti Laptop GPU, 16 GB VRAM (16383 MiB reported by
sd-cli's CUDA query), compute capability 8.6, VMM: yes - GPU driver: Windows WDDM
32.0.15.9174(NVIDIA ≈ 591.74) - Backend used: CUDA — prebuilt
cuda12Windows x64 release,sd-cli/
stable-diffusion.dllreportingcommit d2797b8(master-660-d2797b8) - CUDA runtime DLLs shipped with the release:
cudart64_12.dll,
cublas64_12.dll,cublasLt64_12.dll
The crash is triggered when the CPU backend (the second of the two devices
sd-cli enumerates — #0: CUDA0, #1: CPU) is actually used for tensor
placement / execution.
Steps to reproduce
Run any generation with the model parameters placed on CPU. Minimal form:
sd-cli -M img_gen \
--diffusion-model <model>.gguf \
--vae <vae>.safetensors \
--llm <text-encoder>.gguf \
-p "a cat" -W 512 -H 512 --steps 4 --cfg-scale 1 \
--sampling-method euler -s 42 \
-o out.png \
--offload-to-cpu -v
(I reproduced with a lumina2 / Z-Image-style DiT + a GGUF text encoder, but the
crash is not model-specific — it fires on any module placed on CPU, see below.)
Expected
Image is generated; parameters live in CPU RAM and are streamed to CUDA during
execution (as documented in docs/backend.md).
Actual
Process aborts with STATUS_ILLEGAL_INSTRUCTION. Last log lines before the crash:
[INFO ] ggml_extend_backend.cpp:311 - Found 2 backend devices:
[DEBUG] ggml_extend_backend.cpp:314 - #0: CUDA0
[DEBUG] ggml_extend_backend.cpp:314 - #1: CPU
[DEBUG] ggml_extend_backend.cpp:291 - Initializing backend: CUDA0
[DEBUG] ggml_extend_backend.cpp:291 - Initializing backend: CPU
[DEBUG] <-- crashes here, no further output
So both backends initialize fine; the crash occurs immediately afterward, as
soon as a tensor is loaded onto / run through the CPU backend.
Every CPU-placement form crashes (all on master-660)
| Command fragment | Result |
|---|---|
| (no CPU placement — baseline) | ✅ works |
--offload-to-cpu |
❌ illegal instruction |
--params-backend cpu |
❌ illegal instruction |
--params-backend diffusion=cpu,te=cpu,vae=cpu |
❌ illegal instruction |
--params-backend diffusion=cpu (single module) |
❌ illegal instruction |
--backend vae=cpu (runtime placement, not params) |
❌ illegal instruction |
--backend cuda0 --params-backend cpu (the exact docs/backend.md example) |
❌ illegal instruction |
This shows it is not an argument-parsing problem — even the verbatim example
from docs/backend.md and the legacy --offload-to-cpu flag both crash. The
fault is in the CPU backend code path itself.
Bisection
Same command, same model, same machine:
- master-656-0e4ee04 →
stable-diffusion.cpp:NNNN - generate_image completed in 8.27s✅ - master-660-d2797b8 →
STATUS_ILLEGAL_INSTRUCTION❌
So a change between 656 and 660 broke CPU / multi-backend execution.
Notes / hypotheses
STATUS_ILLEGAL_INSTRUCTIONon a machine without AVX-512 suggests the
prebuilt CPU backend may be emitting an instruction the CPU doesn't support
(e.g. an AVX-512 / VNNI path that isn't guarded by a runtime feature check),
introduced or newly-reached between 656 and 660. It could also be an
out-of-bounds / corrupted dispatch in the new multi-backend graph execution —
I haven't bisected to a specific commit.- The crash is fully reproducible (8/8 runs) on 660 and never reproduces
on 656.
Impact
This blocks all low-VRAM workflows that rely on --offload-to-cpu /
--params-backend cpu — e.g. running a 14B-class video model (WAN 2.2 I2V) on a
16 GB card, which only fits with weights offloaded to RAM. On 660 such a model
either crashes (with offload) or OOM-thrashes (without).
Workaround
Downgrade to master-656-0e4ee04.
I'm happy to test a candidate fix or capture a crash dump / disassembly around
the faulting instruction if that would help narrow it down.
Activity
mpulukkinen commentedon May 31, 2026
Yep, I have the same issue. This is not btw the first time same thing has happened. About couple of weeks ago had the same. Then it got fixed by upgrading to newer version
wbruna commentedon May 31, 2026
It's most likely this Github CI issue: #1483 . That doesn't actually depend on any code change, just on which Github worker built the released binaries.
mpulukkinen commentedon Jun 1, 2026
leejet commentedon Jun 28, 2026
This should be resolved now. Feel free to reopen if it still happens.