[Bug] Vulkan/AMD RX 6950 XT: Flux (split text encoders) silently crashes since master-848-9cdb6b6 — regression vs master-841-6b3edaa (weight preparation OOM)

Summary

Since release master-848-9cdb6b6, sd-cli (official bin-win-vulkan-x64 asset) crashes silently (process exits, no error message on stdout/stderr, bash reports exit code 127) when generating a Flux image with split text encoders on AMD Vulkan. The exact same command works on the previous release asset (master-841-6b3edaa). SDXL with a monolithic checkpoint works fine on the new build, so this looks related to the multi-model loading / weight-residency refactor (candidates: #1940, #1942, #1905).

Environment

OS: Windows 11 (10.0.26200)
GPU: AMD Radeon RX 6950 XT 16 GB (RDNA2 / gfx1030)
Driver: 32.0.21045.5002 (Adrenalin)
Build: sd-master-9cdb6b6-bin-win-vulkan-x64.zip (official release asset), commit 9cdb6b6
Working build for comparison: sd-master-<sha>-bin-win-vulkan-x64.zip with commit 6b3edaa (master-841)

Models (GGUF from the project's recommended sources):

flux1-schnell-Q6_K.gguf (diffusion model)
clip_l.safetensors, t5xxl_fp16.safetensors (text encoders)
ae.safetensors (VAE)

Reproduction

sd-cli.exe --diffusion-model flux1-schnell-Q6_K.gguf \
  --clip_l clip_l.safetensors --t5xxl t5xxl_fp16.safetensors --vae ae.safetensors \
  -p "a red apple on a wooden table, photo" \
  --steps 4 --cfg-scale 1.0 --sampling-method euler -W 512 -H 512 --seed 42 \
  --vae-tiling --backend vulkan -o out.png

Observed on `9cdb6b6` (master-848)

1) Default invocation — silent death. The process terminates without any error message (exit code 127 under Git Bash). Last log lines with -v:

[INFO   ] stable-diffusion.cpp:5322 - get_learned_condition completed, taking 13.73s
[INFO   ] stable-diffusion.cpp:5705 - generating image: 1/1 - seed 42
[INFO   ] ggml_graph_cut.cpp:965  - flux build cached graph cut plan done (taking 6 ms)
[VERBOSE] model_loader.cpp:1243 - loading tensors completed, taking 5.32s (read: 4.06s, ...)
[VERBOSE] model_manager.cpp:428  - model manager prepared params backend buffers (9379.29 MB, 776 tensors, 192 blocks, VRAM) on Vulkan0
<process exits here, no message>

2) --auto-fit off — identical silent crash at the same point.

3) --offload-to-cpu — hard Vulkan OOM error:

ggml_vulkan: vk::Device::createComputePipeline: ErrorOutOfDeviceMemory

4) --max-vram 6 — an actual error line is finally logged, then the process still dies silently:

[ERROR  ] ggml_runner.cpp:848  - t5 segment 17/26 (t5.block.15) failed during weight preparation
<process exits here, exit code 127, no additional message>

Expected / previous behavior

The identical command on the 6b3edaa build completes successfully and produces the image (this GPU is 16 GB; the previous build's streaming/offload behavior fits Flux Q6_K + t5xxl fp16 in VRAM).

Control test: SDXL is unaffected

A monolithic SDXL checkpoint (-m juggernautXL...safetensors, 512×512, euler_a) generates fine on 9cdb6b6 — so the regression is specific to the split-encoders path (diffusion model + separate clip/t5/vae), i.e. exactly what the new model manager / runner lifecycle code (#1940, #1942) and streamed-layer prefetch (#1905) touch.

Additional notes

Two distinct problems, possibly: (a) the weight-preparation memory management regression itself, and (b) the process exits with no diagnostic at all in cases 1/2/4 — even when an [ERROR] line was just logged (case 4), there is no exception message, no abort message, nothing. A graceful failure with the underlying Vulkan error would have made this much easier to report.
No other GPU-heavy process was running; no zombie sd-cli processes; VRAM was free before each run (each configuration was tested in a fresh process).
Happy to test a fix build or additional variants (e.g. quantized t5xxl, flux1-dev, other --max-vram values) on this machine.

docs(veille): sd-cli master-848-9cdb6b6 TESTEE puis ROLLBACK — crash …

Data point: still present in master-849-d04e895 (official bin-win-vulkan-x64 asset).

Tested the exact same Flux schnell command (split text encoders, Vulkan, RX 6950 XT): identical silent crash, exit code 127 with no message, at the same point — right after:

[VERBOSE] model_manager.cpp:428  - model manager prepared params backend buffers (9379.29 MB, 776 tensors, 192 blocks, VRAM) on Vulkan0

Unsurprising since the only commit in 849 (#1939, SDXL embeddings loading) doesn't touch this path. 6b3edaa remains unaffected. I'll keep 6b3edaa installed and can test any candidate fix build on this machine (AMD RX 6950 XT 16 GB, Windows 11, Vulkan).

@laurentvv

@laurentvv, hello, if nobody else will confirm the bug, there is a little possibility that the regression affects only your particular hw+drivers setup and others will not be able to reproduce the error.

If so, since some of 7 commits between 6b3edaa and 9cdb6b6 have binaries in releases, maybe you could try to locate the crashing commit more precisely, to clarify possible reason? (Or are they all crash right after https://github.com/leejet/stable-diffusion.cpp/releases/tag/master-841-6b3edaa)?
Good luck!

@vmobilis

Follow-up per @vmobilis' suggestion — bisected further, the culprit is now isolated to a single commit.

Binary bisect (official bin-win-vulkan-x64 releases, same Flux schnell command as the original report):

Build	Commits added vs `6b3edaa`	Result
master-841 `6b3edaa`	— (reference)	✅ image generated (re-verified 2026-09-08)
master-843 `462d675`	#1905 + #1940	❌ silent crash, exit 127
master-845 `80bac2d`	+ #1941 + #1942	❌ identical crash

Local builds to separate #1905 from #1940 (MSYS2/UCRT64, gcc 16.2, -DGGML_VULKAN=ON, same ggml submodule pin e20c3a14 for all three, same machine/command):

Commit	Result
`6b3edaa` (control)	✅ image generated
`6c57cc3` (#1905 prefetch streamed layers, alone)	✅ image generated — #1905 is not the culprit
`462d675` (#1940 unify runner lifecycles and weight residency)	❌ crash

The regression is introduced by 462d675 — PR #1940 "refactor: unify runner lifecycles and weight residency".

Extra detail that may help: with my gcc build the failure is not fully silent — the process aborts with a visible exception:

what():  vk::Queue::submit: ErrorOutOfDeviceMemory

It happens right after the DiT weights finish streaming to VRAM (last log line model_loader.cpp:1243 - loading tensors completed, i.e. the same stage as the model_manager.cpp:428 buffer-preparation line in the original report). So the ~9.4 GB of DiT buffers get allocated successfully, then the first compute submit runs out of device memory — consistent with the --offload-to-cpu / --max-vram behaviors reported earlier. It looks like the weight-residency refactor changed when (or whether) streamed buffers are released before compute on this 16 GB RDNA2 setup.

Happy to test any candidate fix build on this machine (AMD RX 6950 XT 16 GB, Windows 11, Vulkan, AMD proprietary driver).

master-841-6b3edaa seems to be last working vulkan release. After it, Minimax H3 crashes when loading text encoder.

[INFO ] ggml_graph_cut.cpp:952 - qwen3vl build cached graph cut plan done (taking 4 ms)
\stable-diffusion.cpp\ggml\src\ggml-backend.cpp:930: pre-allocated tensor (text_encoders.llm.model.embed_tokens.weight) in a buffer (Vulkan0) that cannot run the operation (NONE)

I’m unable to reproduce this issue on my device, so I can’t confirm whether #1958 fixes it. Could you give it a try?

Tested leejet's candidate fix PR #1958 (58d3f6e, local MSYS2/UCRT64 build, gcc 16.2,
-DGGML_VULKAN=ON, same ggml submodule pin e20c3a14 as the bisect builds, RX 6950 XT
16 GB, Windows 11, AMD proprietary driver).

The hard crash is fixed — error handling now works as designed — but generation still
does not complete on this card. No more terminate() / vk::Queue::submit: ErrorOutOfDeviceMemory abort on the plain repro: the failure is caught and propagated
cleanly to the CLI with a precise diagnostic. Same command as the original report
(Flux schnell Q6_K, 512×512, 4 steps, euler, --vae-tiling, --backend vulkan):

[WARN   ] model_manager.cpp:1762 - model manager cannot make enough memory available on Vulkan0: need 954.20 MB device / 457.22 MB budget, available 906.93 MB device / unlimited budget
[ERROR  ] ggml_runner.cpp:858  - flux segment 30/59 (flux.single_blocks.9) failed during weight preparation
[ERROR  ] diffusion_engine.cpp:2351 - diffusion model compute failed
[ERROR  ] diffusion_engine.cpp:2462 - Diffusion model sampling failed
[ERROR  ] image.cpp:888  - sampling for image 1/1 failed after 22.02s

Deterministic, not fragmentation: identical failure (same numbers, same segment
30/59) on a freshly rebooted machine with only 0.5/16 GB VRAM in use before launch.
The manager comes up ~47 MB short at flux.single_blocks.9 (need 954.20 MB device,
available 906.93 MB) — mid-sampling, i.e. it can no longer evict enough of the
resident DiT weights to prepare the next segment.
--offload-to-cpu also fails, differently: all 18,858 MB of weights go to RAM as
expected (VRAM 0.00MB, RAM 18858.13MB), loading completes, then the process dies
uncaught with exit 127 during compute:
```
ggml_vulkan: vk::Device::createComputePipeline: ErrorOutOfDeviceMemory
terminate called after throwing an instance of 'vk::OutOfDeviceMemoryError'
  what():  vk::Queue::submit: ErrorOutOfDeviceMemory
```
~15.5 GB of VRAM were free at that point, so a pipeline creation failing with
OutOfDeviceMemory looks like a separate backend/allocation issue on this path
(possibly also refactor: unify runner lifecycles and weight residency #1940-related).

Machine available for any follow-up build. (Also confirmed: master-841 6b3edaa still
generates this exact command fine.)

Could you try running the following command and send me the detailed logs so I can investigate the issue?

sd-cli.exe --diffusion-model flux1-schnell-Q6_K.gguf \
  --clip_l clip_l.safetensors --t5xxl t5xxl_fp16.safetensors --vae ae.safetensors \
  -p "a red apple on a wooden table, photo" \
  --steps 4 --cfg-scale 1.0 --sampling-method euler -W 512 -H 512 --seed 42 \
  --vae-tiling --backend vulkan --offload-to-cpu --log-level debug -o out.png

Ran your exact command on my side — same setup as my earlier reports (AMD RX 6950 XT 16 GB, Windows 11, AMD proprietary driver, uma: 0), local MSYS2/UCRT64 build of 58d3f6e (PR #1958 head, gcc 16.2, -DGGML_VULKAN=ON, ggml pin e20c3a14), with --log-level debug.

Full debug log (192 lines, raw): gist

Result: same uncaught crash as my 2026-09-10 report — exit 127, terminate on vk::Device::createComputePipeline: ErrorOutOfDeviceMemory — but the debug level pinpoints where it happens now: the very first T5 segment, right after a 9 GB "staged compute params" copy into VRAM.

Key excerpt:

[INFO   ] diffusion_engine.cpp:1206 - total params memory size = 18858.13MB (VRAM 0.00MB, RAM 18858.13MB): text_encoders 9318.83MB(RAM), diffusion_model 9379.29MB(RAM), vae 160.00MB(RAM)
[VERBOSE] model_manager.cpp:488  - model manager prepared params backend buffers (235.06 MB, 196 tensors, 1 blocks, RAM) on Vulkan_Host
[VERBOSE] model_manager.cpp:614  - model manager staged compute params (235.06 MB, 196 tensors, 4 blocks) to Vulkan0, taking 0.04s
[DEBUG  ] ggml_runner.cpp:929  - clip executing segment 1/1: graph
[VERBOSE] ggml_runner.cpp:954  - clip compute buffer size: 1.42 MB(VRAM) on Vulkan0 (peak across 1 segment)
[VERBOSE] model_loader.cpp:1279 - loading tensors completed, taking 2.18s (read: 2.02s, memcpy: 0.00s, convert: 0.00s, copy_to_backend: 0.00s)
[VERBOSE] model_manager.cpp:488  - model manager prepared params backend buffers (9083.77 MB, 219 tensors, 10 blocks, RAM) on Vulkan_Host
[VERBOSE] model_manager.cpp:614  - model manager staged compute params (9083.77 MB, 219 tensors, 146 blocks) to Vulkan0, taking 6.05s
[DEBUG  ] ggml_runner.cpp:929  - t5 executing segment 1/1: graph
ggml_vulkan: Compute pipeline creation failed for rms_norm_mul_f32
ggml_vulkan: vk::Device::createComputePipeline: ErrorOutOfDeviceMemory
terminate called after throwing an instance of 'vk::OutOfDeviceMemoryError'
  what():  vk::Queue::submit: ErrorOutOfDeviceMemory

Observations (no interpretation, just what the log shows):

The engine summary counts all 18858.13 MB of weights as RAM, VRAM 0.00 MB — offload to CPU is in effect. Yet right before compute, model_manager "staged compute params (9083.77 MB, 219 tensors, 146 blocks) to Vulkan0" — the full T5 weight set gets copied into VRAM anyway (6.05 s copy). CLIP-L does the same (235 MB staged) and runs fine.
No capacity-guard warning on this path. Without --offload-to-cpu (plain repro), the fix: guard GPU memory capacity and propagate encoding failures #1958 guard logs model manager cannot make enough memory available on Vulkan0… and fails cleanly at flux.single_blocks.9. Here the 9 GB staging to Vulkan0 happens silently and the next pipeline creation OOMs uncaught — the guard doesn't seem to cover the staging step.
Puzzling detail: with VRAM at 0.7/16 GB before launch and ~9.3 GB staged in total, roughly 6 GB should still be free, yet createComputePipeline for rms_norm_mul_f32 already fails with ErrorOutOfDeviceMemory. Same deterministic behavior as before (rebooted-machine test on 2026-09-10), max_vram: 0 in the params dump.
No image written (failure occurs during text-encoding, before sampling).

Happy to test any follow-up build on this machine.

📄 Full raw debug log (192 lines — 12 terminal progress-bar redraw lines elided for readability; byte-exact raw version in the gist linked above)

# ============================================================
# Test demande par leejet — issue #1946, commentaire 5636742869 (2026-09-11)
# Binaire   : build local MSYS2/UCRT64 gcc 16.2 -DGGML_VULKAN=ON de 58d3f6e (PR #1958 head), ggml pin e20c3a14
# Materiel  : AMD RX 6950 XT 16 Go, Windows 11, pilote AMD proprietaire
# Etat VRAM avant lancement : 0.7/16 Go utilises (machine sans autre charge GPU)
# Commande  : commande exacte de leejet, chemins absolus
# ============================================================
[VERBOSE] main.cpp:700  - version: stable-diffusion.cpp version unknown, commit unknown
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 6950 XT (AMD proprietary driver) | uma: 0 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
[VERBOSE] main.cpp:701  - System Info: 
   SSE3 = 1 |    OPENMP = 1 |    REPACK = 1 | 
[VERBOSE] main.cpp:702  - SDCliParams {
  mode: img_gen,
  output_path: "C:\GIT\generator-assets\scratch\bisect_1946\out_leejet_debug.png",
  image_path: "",
  metadata_format: "text",
  log_level: debug,
  color: false,
  canny_preprocess: false,
  convert_name: false,
  preview_method: none,
  preview_interval: 1,
  preview_path: "preview.png",
  preview_fps: 16,
  taesd_preview: false,
  preview_noisy: false,
  imatrix_out: "",
  metadata_raw: false,
  metadata_brief: false,
  metadata_all: false
}
[VERBOSE] main.cpp:703  - SDContextParams {
  n_threads: 12,
  model_path: "",
  clip_l_path: "C:\Modeles_LLM\clip_l.safetensors",
  clip_g_path: "",
  clip_vision_path: "",
  t5xxl_path: "C:\Modeles_LLM\t5xxl_fp16.safetensors",
  llm_path: "",
  llm_vision_path: "",
  diffusion_model_path: "C:\Modeles_LLM\flux1-schnell-Q6_K.gguf",
  high_noise_diffusion_model_path: "",
  uncond_diffusion_model_path: "",
  embeddings_connectors_path: "",
  vae_path: "C:\Modeles_LLM\ae.safetensors",
  vae_format: "auto",
  audio_vae_path: "",
  taesd_path: "",
  esrgan_path: "",
  control_net_path: "",
  embedding_dir: "",
  embeddings: {
  }
  wtype: NONE,
  tensor_type_rules: "",
  lora_model_dir: ".",
  hires_upscalers_dir: "",
  photo_maker_path: "",
  rng_type: cuda,
  sampler_rng_type: NONE,
  offload_params_to_cpu: true,
  max_vram: "0",
  disable_prefetch: false,
  disable_segmented_compute: false,
  eager_load: false,
  backend: "vulkan",
  params_backend: "",
  split_mode: "",
  model_args: "",
  auto_fit: true,
  enable_mmap: false,
  control_net_cpu: false,
  clip_on_cpu: false,
  vae_on_cpu: false,
  flash_attn: false,
  diffusion_flash_attn: false,
  diffusion_conv_direct: false,
  vae_conv_direct: false,
  prediction: NONE,
  lora_apply_mode: auto,
  force_sdxl_vae_conv_scale: false
}
[VERBOSE] main.cpp:704  - SDGenerationParams {
  loras: "{
  }",
  high_noise_loras: "{
  }",
  prompt: "a red apple on a wooden table, photo",
  negative_prompt: "",
  ad_model_path: "",
  ad_prompt: "",
  ad_negative_prompt: "",
  extra_ad_args: "",
  clip_skip: -1,
  width: 512,
  height: 512,
  batch_count: 1,
  qwen_image_layers: 3,
  init_image_path: "",
  end_image_path: "",
  mask_image_path: "",
  control_image_path: "",
  ref_image_paths: [],
  ref_video_paths: [],
  ref_video_audio_paths: [],
  ref_audio_paths: [],
  control_video_path: "",
  auto_resize_ref_image: true,
  increase_ref_index: false,
  pm_id_images_dir: "",
  pm_id_embed_path: "",
  pm_style_strength: 20,
  skip_layers: [7, 8, 9],
  sample_params: (txt_cfg: 1.00, img_cfg: 1.00, distilled_guidance: 3.50, slg.layer_count: 0, slg.layer_start: 0.01, slg.layer_end: 0.20, slg.scale: 0.00, scheduler: NONE, sample_method: euler, sample_steps: 4, eta: inf, shifted_timestep: 0, flow_shift: inf, extra_sample_args: ),
  high_noise_skip_layers: [7, 8, 9],
  high_noise_sample_params: (txt_cfg: 7.00, img_cfg: 7.00, distilled_guidance: 3.50, slg.layer_count: 0, slg.layer_start: 0.01, slg.layer_end: 0.20, slg.scale: 0.00, scheduler: NONE, sample_method: NONE, sample_steps: 20, eta: inf, shifted_timestep: 0, flow_shift: inf, extra_sample_args: ),
  custom_sigmas: [],
  cache_mode: "",
  cache_option: "",
  cache: disabled (threshold=inf, start=0.15, end=0.95),
  moe_boundary: 0.875,
  video_frames: 1,
  fps: 16,
  vace_strength: 1,
  strength: 0.75,
  control_strength: 0.9,
  seed: 42,
  upscale_repeats: 1,
  upscale_tile_size: 128,
  hires: { enabled: false, upscaler: "Latent", model_path: "", scale: 2, target_width: 0, target_height: 0, steps: 0, denoising_strength: 0.7, custom_sigmas: [], upscale_tile_size: 128 },
  vae_tiling_params: { 1, 0, 0, 0, 0.5, 0, 0, "" },
}
[VERBOSE] model_loader.cpp:174  - using 12 threads for model loading
[INFO   ] diffusion_engine.cpp:700  - loading diffusion model from 'C:\Modeles_LLM\flux1-schnell-Q6_K.gguf'
[INFO   ] model_loader.cpp:191  - load C:\Modeles_LLM\flux1-schnell-Q6_K.gguf using gguf format
[VERBOSE] model_loader.cpp:237  - init from 'C:\Modeles_LLM\flux1-schnell-Q6_K.gguf'
[INFO   ] diffusion_engine.cpp:721  - loading clip_l from 'C:\Modeles_LLM\clip_l.safetensors'
[INFO   ] model_loader.cpp:197  - load C:\Modeles_LLM\clip_l.safetensors using safetensors format
[VERBOSE] model_loader.cpp:265  - init from 'C:\Modeles_LLM\clip_l.safetensors', prefix = 'clip_l.'
[INFO   ] diffusion_engine.cpp:742  - loading t5xxl from 'C:\Modeles_LLM\t5xxl_fp16.safetensors'
[INFO   ] model_loader.cpp:197  - load C:\Modeles_LLM\t5xxl_fp16.safetensors using safetensors format
[VERBOSE] model_loader.cpp:265  - init from 'C:\Modeles_LLM\t5xxl_fp16.safetensors', prefix = 'text_encoders.t5xxl.transformer.'
[INFO   ] diffusion_engine.cpp:771  - loading vae from 'C:\Modeles_LLM\ae.safetensors'
[INFO   ] model_loader.cpp:197  - load C:\Modeles_LLM\ae.safetensors using safetensors format
[VERBOSE] model_loader.cpp:265  - init from 'C:\Modeles_LLM\ae.safetensors', prefix = 'vae.'
[INFO   ] diffusion_engine.cpp:893  - Version: Flux 
[VERBOSE] ggml_extend_backend.cpp:395  - Initializing backend: Vulkan0
[VERBOSE] ggml_extend_backend.cpp:395  - Initializing backend: CPU
[VERBOSE] model_loader.cpp:174  - using 12 threads for model loading
[INFO   ] diffusion_engine.cpp:1385 - Weight type stat:                      f32: 712  |     f16: 419  |    q6_K: 304  
[INFO   ] diffusion_engine.cpp:1386 - Conditioner weight type stat:          f16: 415  
[INFO   ] diffusion_engine.cpp:1387 - Diffusion model weight type stat:      f32: 468  |     f16: 4    |    q6_K: 304  
[INFO   ] diffusion_engine.cpp:1388 - VAE weight type stat:                  f32: 244  
[VERBOSE] diffusion_engine.cpp:1390 - ggml tensor size = 432 bytes
[VERBOSE] clip_tokenizer.cpp:66   - vocab size: 49408
[VERBOSE] flux.hpp:185  - flux: depth = 19, depth_single_blocks = 38, guidance_embed = false, context_in_dim = 4096, hidden_size = 3072, num_heads = 24
[INFO   ] model_builders.cpp:505  - using VAE for encoding / decoding
[INFO   ] auto_encoder_kl.hpp:529  - vae decoder: ch = 128
[VERBOSE] diffusion_engine.cpp:1101 - validating model metadata
[VERBOSE] diffusion_engine.cpp:1160 - model metadata validated; weights will be prepared lazily
[INFO   ] diffusion_engine.cpp:1206 - total params memory size = 18858.13MB (VRAM 0.00MB, RAM 18858.13MB): text_encoders 9318.83MB(RAM), diffusion_model 9379.29MB(RAM), vae 160.00MB(RAM), controlnet 0.00MB(N/A), extensions 0.00MB(N/A)
[INFO   ] diffusion_engine.cpp:1340 - running in Flux FLOW mode
[INFO   ] image.cpp:798  - generate_image 512x512
[VERBOSE] model_loader.cpp:174  - using 12 threads for model loading
[INFO   ] diffusion_engine.cpp:2719 - Using 'flux_kontext' preset for reference images
[INFO   ] denoiser.hpp:1125 - get_sigmas with Flux scheduler
[VERBOSE] denoiser.hpp:765  - Flux scheduler: image_seq_len=4096, steps=4, mu=1.150
[INFO   ] request.cpp:415  - sampling using Euler method
[VERBOSE] conditioner.hpp:1139 - parse 'a red apple on a wooden table, photo' to [['a red apple on a wooden table, photo', 1], ]
[VERBOSE] bpe_tokenizer.cpp:208  - split prompt "a red apple on a wooden table, photo" to 9 tokens ["a</w>", "red</w>", "apple</w>", "on</w>", "a</w>", "wooden</w>", "table</w>", ",</w>", "photo</w>", ]
[VERBOSE] t5_unigram_tokenizer.cpp:336  - split prompt "a red apple on a wooden table, photo" to tokens ["▁", "a", "▁red", "▁apple", "▁on", "▁", "a", "▁wooden", "▁table", ",", "▁photo", ]
[VERBOSE] clip.hpp:326  - identity projection
[INFO   ] ggml_graph_cut.cpp:963  - clip build cached graph cut plan done (taking 0 ms)
[VERBOSE] model_loader.cpp:1021 - loading 196/196 tensors from C:\Modeles_LLM\clip_l.safetensors

[INFO   ] model_loader.cpp:1279 - loading tensors completed, taking 0.20s (read: 0.05s, memcpy: 0.00s, convert: 0.00s, copy_to_backend: 0.00s)
[VERBOSE] model_manager.cpp:488  - model manager prepared params backend buffers (235.06 MB, 196 tensors, 1 blocks, RAM) on Vulkan_Host
[VERBOSE] model_manager.cpp:614  - model manager staged compute params (235.06 MB, 196 tensors, 4 blocks) to Vulkan0, taking 0.04s
[DEBUG  ] ggml_runner.cpp:929  - clip executing segment 1/1: graph
[VERBOSE] ggml_runner.cpp:954  - clip compute buffer size: 1.42 MB(VRAM) on Vulkan0 (peak across 1 segment)
[INFO   ] ggml_graph_cut.cpp:963  - t5 build cached graph cut plan done (taking 1 ms)
[VERBOSE] model_loader.cpp:1021 - loading 219/219 tensors from C:\Modeles_LLM\t5xxl_fp16.safetensors

[INFO   ] model_loader.cpp:1279 - loading tensors completed, taking 2.18s (read: 2.02s, memcpy: 0.00s, convert: 0.00s, copy_to_backend: 0.00s)
[VERBOSE] model_manager.cpp:488  - model manager prepared params backend buffers (9083.77 MB, 219 tensors, 10 blocks, RAM) on Vulkan_Host
[VERBOSE] model_manager.cpp:614  - model manager staged compute params (9083.77 MB, 219 tensors, 146 blocks) to Vulkan0, taking 6.05s
[DEBUG  ] ggml_runner.cpp:929  - t5 executing segment 1/1: graph
ggml_vulkan: Compute pipeline creation failed for rms_norm_mul_f32
ggml_vulkan: vk::Device::createComputePipeline: ErrorOutOfDeviceMemory
terminate called after throwing an instance of 'vk::OutOfDeviceMemoryError'
  what():  vk::Queue::submit: ErrorOutOfDeviceMemory

Thanks for testing. The log narrows the failure to T5’s first Vulkan graph execution, after its weights have been staged successfully.

I found a cleanup path that can throw a second exception while handling the pipeline creation failure. That explains why the final error mentions Queue::submit and the new error handler does not get a chance to report it.

For the original OOM, I suspect the smaller weight allocations introduced by the refactor may interact with Vulkan’s preference for CPU-visible VRAM. Could you run the same build and command twice from MSYS2 Bash:

GGML_VK_MEMORY_LOGGER=1 <same sd-cli command>

GGML_VK_MEMORY_LOGGER=1 GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM=1 <same sd-cli command>

Please attach both logs and, if available, the memory heaps/types section from vulkaninfo. This should help distinguish memory placement pressure from a pipeline-specific driver issue.

Both runs done, same setup as my debug log (RX 6950 XT 16 GB, Windows 11, AMD proprietary
driver, local 58d3f6e build, exact same command: Flux schnell Q6_K, --offload-to-cpu --log-level debug).

The headline: GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM=1 makes the whole --offload-to-cpu
path work. Clean exit 0, correct image (red apple on wooden table, seed 42). Same 9,083.77
MB T5 staging that died at pipeline creation now completes — and everything is faster, not
just working: T5 staging 1.25 s (vs 5.84 s in run 1), conditioning 4.71 s, sampling 11.65 s
(4 steps), VAE decode 1.41 s, image saved.

Run 1 (GGML_VK_MEMORY_LOGGER=1 only) pins the failure exactly where you suspected — T5's
first graph execution, after successful staging, and with the double exception you described
(the terminate reports Queue::submit while the failing call was createComputePipeline):

[VERBOSE] model_manager.cpp:614  - model manager staged compute params (9083.77 MB, 219 tensors, 146 blocks) to Vulkan0, taking 5.84s
ggml_vulkan memory: ggml_backend_vk_buffer_type_alloc_buffer(71566344)
ggml_vulkan memory: Vulkan0: +68.25 MiB device at 0x152096f9310. Total device: 9.17 GiB, total host: 9.10 GiB
[DEBUG   ] ggml_runner.cpp:929  - t5 executing segment 1/1: graph
ggml_vulkan: Compute pipeline creation failed for rms_norm_mul_f32
ggml_vulkan: vk::Device::createComputePipeline: ErrorOutOfDeviceMemory
terminate called after throwing an instance of 'vk::OutOfDeviceMemoryError'
  what():  vk::Queue::submit: ErrorOutOfDeviceMemory

Note the sequence: the 68.25 MiB compute buffer allocates fine right after staging, then the
next allocation (pipeline creation) OOMs while the logger reports 9.17 GiB total device —
with roughly 6 GB of the 16 GB card still free.

vulkaninfo memory layout for this card (full section in the gist) matches your CPU-visible
VRAM hypothesis:

heap 0: 15.73 GiB DEVICE_LOCAL (+MULTI_INSTANCE) — real VRAM
heap 1: 15.92 GiB, no flags — shared/GTT
heap 2: 256 MiB DEVICE_LOCAL|HOST_VISIBLE — the host-visible VRAM window
type 2 = DEVICE_LOCAL|HOST_VISIBLE|HOST_COHERENT → heap 2 (256 MiB)

So the only host-visible device-local pool this driver exposes is 256 MiB. With host-visible
vidmem enabled, device allocations (staged weights, pipeline resources) interact with that
tiny window and pipeline creation ends up ErrorOutOfDeviceMemory despite heap 0 being
mostly free; disabling it forces plain device-local placement and the path both works and
gets measurably faster.

Full logs, raw (single gist, 3 files): https://gist.github.com/laurentvv/6422321845c3466179f20f7ec5c0cf80

run1_memlogger_crash.log — GGML_VK_MEMORY_LOGGER=1 (exit 127)
run2_nohostvisible_SUCCESS.log — +GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM=1 (exit 0, full memory-logger output)
vulkaninfo_memory_heaps_types.txt — heaps/types section for this GPU

Happy to A/B any candidate (e.g. gating host-visible usage by the heap size, or defaulting it
off when a small host-visible device-local heap is detected).

Also tested your --max-vram suggestion (from #1947) on my #1946 repro — plain path, i.e. no --offload-to-cpu and no GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM, so this maps to the original (non-offload) failure. Same setup as my earlier reports (RX 6950 XT 16 GB, Windows 11, AMD proprietary driver, local 58d3f6e build = PR #1958 head; Flux schnell Q6_K, 512×512, 4 steps, euler, seed 42, --vae-tiling), --max-vram 10/12/14 (GiB).

The graph still doesn't complete on this card, but --max-vram produces a very clean diagnostic. The budget mechanism itself works: with a cap set, reported free device memory tracks the cap exactly (the underflowed free-VRAM accounting from my original report is bypassed). What fails instead is the budget-side bookkeeping: the manager fills residency up to the cap, then can no longer evict enough to make budget headroom for the next segment's streaming weights.

`--max-vram`	fails at	need (device / budget)	available (device / budget)
(none, unlimited)	segment 30/59 `flux.single_blocks.9`	954.20 MB / 457.22 MB	906.93 MB / unlimited
10 GiB	segment 4/59 `flux.double_blocks.2`	793.13 MB / 537.55 MB	6187.66 MB / 316.08 MB
12 GiB	segment 11/59 `flux.double_blocks.9`	793.13 MB / 537.55 MB	4324.72 MB / 501.13 MB
14 GiB	segment 19/59 `flux.double_blocks.17`	793.13 MB / 537.55 MB	2195.64 MB / 420.06 MB

Observations:

With a budget set, the failing need is constant (793.13 MB device / 537.55 MB budget — streamed weights + runtime peak, identical at every death point), while physical device memory is plentiful (the cap still leaves 2.2–6.2 GB free). The only blocker is budget headroom: once residency saturates the cap, the eviction pass never frees the 537.55 MB of budget needed for staging. The death segment moves later as the budget grows (4 → 11 → 19) simply because more fits, but every value dies the same way.
Best case (12 GiB) misses by only 36.42 MB of budget (need 537.55 / available 501.13).
Without a budget, the per-segment budget need is lower (457.22 MB at segment 30 — single blocks are lighter than double blocks), but there it's the broken free-VRAM report that blocks instead (need 954.20 / available 906.93 device).

So on this card (18,858 MB of weights vs 16 GB VRAM — stream-through is mandatory), no --max-vram value I tried lets the graph complete; the failure just moves. The eviction/budget interplay on the stream-through path looks like the remaining blocker for the plain (non-offload) path — happy to test any candidate that changes it.

Full logs (10/12/14 GiB): gist

I made an attempt to fix the issue. Please try the latest code from #1958.

Run the following command and check whether the issue has been resolved. If the problem still persists, please send me the detailed debug logs so I can investigate further.

sd-cli.exe --diffusion-model flux1-schnell-Q6_K.gguf \
  --clip_l clip_l.safetensors --t5xxl t5xxl_fp16.safetensors --vae ae.safetensors \
  -p "a red apple on a wooden table, photo" \
  --steps 4 --cfg-scale 1.0 --sampling-method euler -W 512 -H 512 --seed 42 \
  --vae-tiling --backend vulkan --offload-to-cpu --log-level debug -o out.png

Tested the latest code from PR #1958 (head 1bf76e6, including da4e841 "handle backend cleanup errors and use 256 MiB residency blocks" and 5c7fc62 "use 1 GiB residency blocks") on the same setup as my previous reports.

Setup: RX 6950 XT 16 GB, Windows 11, AMD proprietary driver, local MSYS2/UCRT64 gcc build of PR head 1bf76e6, ggml submodule at e20c3a14 (unchanged from my bisect — conditions identical to my earlier tests). Flux schnell Q6_K, 512×512, 4 steps, euler, seed 42, --vae-tiling --backend vulkan.

Results — both failure paths from my previous reports now work:

Plain path (no offload, the original [Bug] Vulkan/AMD RX 6950 XT: Flux (split text encoders) silently crashes since master-848-9cdb6b6 — regression vs master-841-6b3edaa (weight preparation OOM) #1946 failure): ✅ fixed. Generation completes (generate_image completed in 13.14s, exit 0). Previously (58d3f6e) it failed deterministically at segment 30/59 (flux.single_blocks.9), ~47 MB short, and on master ≥848 it silently crashed.
--offload-to-cpu (your suggested command, with --log-level debug): ✅ works too. Exit 0, image saved (completed in 17.41s). Previously (58d3f6e) this path died with a silent exit 127 on vk::Device::createComputePipeline: ErrorOutOfDeviceMemory despite ~15.5 GB free VRAM.

Outputs (repro prompt "a photo of a red fox in snow" for #1, your apple command for #2):

Sanity check vs pre-regression baseline: same repro on the installed 6b3edaa (master-841) binary produces a visually identical image — mean per-pixel difference 1.6/255 (small numeric drift expected from the merged master refactors):

One extra observation: with ~2-3 GB of VRAM occupied by another process, the old 6b3edaa binary fails at flux prepare (OOM on a 331 MB params buffer) while the PR build had just completed the same run — the new residency manager also seems more robust under VRAM pressure on this 16 GB card.

Thanks — the residency block sizing was indeed the missing piece. From my side the regression is resolved; happy to re-test if you change anything else before merging.

Thanks for testing. Since all the issues have been fixed, I’ll merge the corresponding PR.

Navigation Menu

Repository navigation

[Bug] Vulkan/AMD RX 6950 XT: Flux (split text encoders) silently crashes since master-848-9cdb6b6 — regression vs master-841-6b3edaa (weight preparation OOM) #1946

Summary

Environment

Reproduction

Observed on `9cdb6b6` (master-848)

Expected / previous behavior

Control test: SDXL is unaffected

Additional notes

Happy to test any follow-up build on this machine.

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[Bug] Vulkan/AMD RX 6950 XT: Flux (split text encoders) silently crashes since master-848-9cdb6b6 — regression vs master-841-6b3edaa (weight preparation OOM) #1946

Description

Summary

Environment

Reproduction

Observed on 9cdb6b6 (master-848)

Expected / previous behavior

Control test: SDXL is unaffected

Additional notes

Activity

laurentvv commented on Sep 7, 2026

vmobilis commented on Sep 8, 2026

laurentvv commented on Sep 8, 2026

mlsterpr0 commented on Sep 9, 2026

leejet commented on Sep 10, 2026

laurentvv commented on Sep 10, 2026

leejet commented on Sep 11, 2026

laurentvv commented on Sep 11, 2026

Happy to test any follow-up build on this machine.

leejet commented on Sep 11, 2026

laurentvv commented on Sep 11, 2026

laurentvv commented on Sep 11, 2026

leejet commented on Sep 13, 2026

laurentvv commented on Sep 13, 2026

leejet commented on Sep 13, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Observed on `9cdb6b6` (master-848)