Skip to content

[Bug] ggml commit cc937ae causes Krea2 generation to OOM on AMD dGPU #2254

Closed

Description

@ThGrSoRu

Git commit

stable-diffusion.cpp at c6dfa9d and ggml at c57b71a

Operating System & Version

Linux 7.3.0-gc339f4010dea-dirty

GGML backends

Vulkan

Command-line arguments used

sd-cli -M img_gen --diffusion-model /home/max/ComfyUI/models/diffusion_models/kroma-0.3.1-turbo-opd-Q8_0.gguf --llm /home/max/ComfyUI/models/text_encoders/qwen-3-vl-4B-Q8_0.gguf --vae /home/max/ComfyUI/models/vae/qwen-BF16.safetensors --lora-model-dir /home/max/ComfyUI/models/loras/ -p "$(cat /home/media/Samsung_SSD_970_Evo/text/machines/prompt_pos.txt)" --conv-direct on --flash-attn on --mmap --offload-to-cpu --rng cpu --cfg-scale 1 --sampling-method euler --scheduler discrete --steps 8 -H 1184 -W 888 -s 614094079319347 -o (date --rfc-3339=second).png

Steps to reproduce

I compiled stable.cpp at c6dfa9d and its ggml fork at c57b71a, and have observed warnings, errors and failures during image generation right away.
These were somewhat non-deterministic between different image generations of the same workflow and the same commit, but are definitely attributable to and introduced by cc937ae.

I managed to make this a lot more reproducible by running Minecraft in the background, which used about 2GiB of VRAM.
With this, running my workflow at the specifiec commits have always resulted in a failure quite early, at the memory mapping stage or the first few sampling steps:

  |==================>                               | 3/8 -  11.16s/it - ETA 01:06[W] model manager memory on Vulkan0: reported free 117.54 MB / total 12272.00 MB, tracked weights 1462.25 MB / other runtime 0.00 MB / current runtime 1296.77 MB --- model_manager.cpp:1794
[W] model manager cannot make enough memory available on Vulkan0: need 512.00 MB device / 1296.77 MB budget, available 117.54 MB device / unlimited budget --- model_manager.cpp:2024
[E] krea2 segment 22/29 (krea2.blocks.21) failed during allocated capacity check --- ggml_runner.cpp:933
[E] diffusion model compute failed --- diffusion_engine.cpp:2655
[E] Diffusion model sampling failed --- diffusion_engine.cpp:2888
[E] sampling for image 1/1 failed after 48.15s --- image.cpp:961
[E] generate failed --- main.cpp:926

However, running stable.cpp with ggml at 6a89493 and with Minecraft in the background, did not result in a single warning, error or failure on multiple runs of the same workflow.

Prepending GGML_VK_REUSE_BUFFERS=0, introduced by cc937ae, did not help.

What you expected to happen

The opposite of the above.

What actually happened

See above.

Logs / error messages / stack trace

No response

Additional context / environment details

apiVersion        = 1.4.363 (4211051)
driverVersion     = 26.99.99 (109457507)
vendorID          = 0x1002
deviceID          = 0x73df
deviceType        = PHYSICAL_DEVICE_TYPE_DISCRETE_GPU
deviceName        = AMD Radeon RX 6700 XT (RADV NAVI22)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions