Repository navigation
Make a PR updating ggml and support CUDA backend #72
Description
@leejet I intend to create a pull request that requires me to use the latest version of ggml to utilize ggml-alloc and ggml-backend for adding GPU acceleration to this project. The issue is that I need some feedback to make progress. I'm not sure if you're already working on something to avoid redoing tasks that are already done.
Reacted by Jonathan Frank, lin72h, JohnClaw, Erik Scholz, Weirui Kuang and LeonNerd
Activity
[-]Why don't use ggml-alloc instead dynamic mode that you implemented?[/-][+]Make a PR updating ggml and support CUDA backend[/+]FSSRepo commentedon Oct 17, 2023
@Green-Sky I forked this project, but I want to use a more recent version of ggml, precisely the fork of ggml that I already have, where I've implemented CUDA acceleration for Conv2D operation, and I also want to replace the way tensor memory is managed using ggml-alloc. The problem I'm facing is that I don't know how to replace the submodule pointing to leejet/ggml with FSSRepo/ggml. Can you help me or provide any suggestions?
Green-Sky commentedon Oct 17, 2023
afaik leejet has some other custom changes. or at least had last time i checked.
the easiest way would be to navigate to the submodule in the filesystem,
type
git remote add fssrepo <your_fork_url.git>then
git fetch --alland then checkout a branch of yours. if the branch name is duplicated its a bit ugly, it will be something likefssrepo/branchname.FSSRepo commentedon Oct 17, 2023
I will try it. Thank you!
leejet commentedon Oct 22, 2023
I'm glad to hear this news. Although I have plans to add GPU support, I haven't started working on it yet. I've been quite busy with work-related matters recently. Looking forward to your progress.
leejet commentedon Oct 22, 2023
And just now, I've synced the latest changes from GGML's upstream to leejet/ggml. You can make your modifications based on this branch https://github.com/leejet/ggml/tree/sd.cpp. The only difference from the latest GGML is the addition of the dynamic mode.
FSSRepo commentedon Oct 22, 2023
I had done the changes from your dynamic mode to
ggml-allocbut the compute graph waste too much memoryFor a image 512 x 512 (diffusion compute):
Dynamic mode: 560 MB
ggml-alloc: 1147 MB
Vae decoding compute:
Dynamic mode: ~1700 MB
ggml-alloc: 2496 MB fixed
ggml alloc is needed for implement differents backends
FSSRepo commentedon Oct 22, 2023
The latest commit crashes in my computer (after sync ggml):
Before sync ggml (works):
For some reason ggml crashes when call
ggml_build_forwardin any program. Usingggml_build_forward_expandthis is not ocurr.Debug in Visual Studio
your ggml version is broken
leejet commentedon Oct 23, 2023
This is because the latest ggml increased the size of 'struct ggml_cplan,' which caused a stack overflow issue on MSVC. I've just submitted a commit to fix this problem. You can give it a try.
FSSRepo commentedon Nov 26, 2023
Closed by #75