You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It may be possible to find an optimal setting by testing different combinations of openblas threads and --threads. I guess this also applies to llama.cpp when using openblas.
It may be possible to find an optimal setting by testing different combinations of openblas threads and --threads. I guess this also applies to llama.cpp when using openblas.
Because ggml's thread is always busy waiting, even if no computation task is performed. As a result, it competes with blas threads, sometimes resulting in negative optimization. This is the point that needs to be optimized.
This is expected as ggml conv 2d will not be optimized by blas and will not run on GPU, I am working on this issue
Btw, I'm not sure if the CPU version of conv 2d is optimal - most likely it is not.
There might be additional improvements possible if implemented properly
GPU support is already in my TODO list, and I'm working on adding it. However, I need to make ggml_conv_2d works on the GPU first.
Great. But using cuBLAS is currently better than anything else if you have a cuda GPU.
Because ggml's thread is always busy waiting, even if no computation task is performed. As a result, it competes with blas threads, sometimes resulting in negative optimization. This is the point that needs to be optimized.
Dont know if OpenBlas will have any real benefits over building without blas. And it looks like support for OpenBlas will soon be removed. See ggml-org/llama.cpp#2372
@klosax I suspect something is off with your benchmarks because it seems like the speed gain should be much higher with GPU vs CPU. Am I wrong?
Using -DGGML_CLBLAST and applying the patch provided by ggml creator in #48, the GPU does get activated BUT CPU temps don’t drop by much. GPU stays hot after completion, so maybe those flags are not really doing anything.
Did yesterday’s cuda update affect your speed at all?
so,i have a question, the project can only run on the cpu?i found vram is low when use cuBLAS by enabling it:
-DGGML_CUBLAS=ON and It has been a long time when use FP16 model.
what should i do for better infer time.
@LeonNerd I think the main reason of this project exist is can run with CPU only + low RAM. But when we look at README.md, GPU inference is still on development.
And for shorter generation time, well maybe just generate 512 x 512 image only(don't use any BLAS), or get better CPU(?)
Activity
Green-Sky commentedon Aug 15, 2023
did you run it/benchmark it?
klosax commentedon Aug 15, 2023
Tests on 6-core Ryzen 5 and GTX 1660
Parameter
--threadsis set to physical cores.cuBLAS is fastest when using F16 model and about 4.5s (22%) faster than without blas
OpenBLAS is only faster (1.6s) than without blas when using F32 model and setting openblas threads to physical cores.
Time for one sampling step:
Time for decode_first_stage stays at about 56s in all tests.
OpenBLAS environment variable to set number of threads:
export OPENBLAS_NUM_THREADS=6klosax commentedon Aug 15, 2023
It may be possible to find an optimal setting by testing different combinations of openblas threads and --threads. I guess this also applies to llama.cpp when using openblas.
leejet commentedon Aug 16, 2023
GPU support is already in my TODO list, and I'm working on adding it. However, I need to make ggml_conv_2d works on the GPU first.
leejet commentedon Aug 16, 2023
This is expected as ggml conv 2d will not be optimized by blas and will not run on GPU, I am working on this issue
leejet commentedon Aug 16, 2023
Because ggml's thread is always busy waiting, even if no computation task is performed. As a result, it competes with blas threads, sometimes resulting in negative optimization. This is the point that needs to be optimized.
ggerganov commentedon Aug 16, 2023
Btw, I'm not sure if the CPU version of conv 2d is optimal - most likely it is not.
There might be additional improvements possible if implemented properly
klosax commentedon Aug 16, 2023
Great. But using cuBLAS is currently better than anything else if you have a cuda GPU.
Dont know if OpenBlas will have any real benefits over building without blas. And it looks like support for OpenBlas will soon be removed. See ggml-org/llama.cpp#2372
Happenedtostumblein commentedon Sep 5, 2023
@klosax I suspect something is off with your benchmarks because it seems like the speed gain should be much higher with GPU vs CPU. Am I wrong?
Using -DGGML_CLBLAST and applying the patch provided by ggml creator in #48, the GPU does get activated BUT CPU temps don’t drop by much. GPU stays hot after completion, so maybe those flags are not really doing anything.
Did yesterday’s cuda update affect your speed at all?
LeonNerd commentedon Sep 12, 2023
so,i have a question, the project can only run on the cpu?i found vram is low when use cuBLAS by enabling it:
-DGGML_CUBLAS=ON and It has been a long time when use FP16 model.
what should i do for better infer time.
juniofaathir commentedon Sep 12, 2023
@LeonNerd I think the main reason of this project exist is can run with CPU only + low RAM. But when we look at README.md, GPU inference is still on development.
And for shorter generation time, well maybe just generate 512 x 512 image only(don't use any BLAS), or get better CPU(?)
LeonNerd commentedon Sep 15, 2023
Can we get some inspiration from clip.cpp?
FSSRepo commentedon Nov 29, 2023
@LeonNerd You can now activate the CUDA backend with
-DSD_CUBLAS=ON, @klosax you can close this issue.Green-Sky commentedon Nov 29, 2023
@klosax would be cool if you could rerun the benchmarks 😉
feat: add VAE encoding tiling support and adaptive overlap (#484)
leejet commentedon Sep 14, 2025
Support for CUDA GPU acceleration has already been added.
Squashed 'ggml/' changes from e20c3a1..04b55bba