Skip to content

Ship current Vulkan prebuilts and restore Volta CUDA support - #1

Open
oobabooga wants to merge 3 commits into
masterfrom
fix/h3-native-prebuilts
Open

oobabooga wants to merge 3 commits into
masterfrom
fix/h3-native-prebuilts

Conversation

@oobabooga

Copy link
Copy Markdown
Owner

MiniMax-H3's Vulkan path can crash during denoising on Linux or abort while loading the text encoder on Windows. The fixes are already in this repository, but Studio gets an older upstream Vulkan build because this workflow does not publish Vulkan assets.

The Linux CUDA prebuilt also has no sm70 kernels, so it cannot run on a V100 even though the device probe can list the card.

Root cause

The older Vulkan build failed in two different places on the AMD runners:

  • On Linux, the scheduler rewrites graph pointers during a mixed CPU/Vulkan segment. The next segment reuses those pointers after the scheduler has freed their tensors, causing a segfault.
  • On Windows, the embedding type allowlist excludes BF16. The encoder's embedding tensor is expanded to a 3.11 GB F32 buffer, which exceeds the driver's buffer-size limit.

Current source restores the graph state between segments and keeps supported BF16 embedding weights in BF16. This PR packages those fixes for Vulkan.

Fix

  • Build sd-cli and sd-server for Linux x86_64 Vulkan and Windows x64 Vulkan. The Windows binaries go through the existing signing steps.
  • Build Linux on Ubuntu 22.04. Studio does not check glibc when selecting an asset, so a 24.04 build could be installed on a host that cannot load it.
  • Use Vulkan SDK 1.4.328.1, with both downloads checked against pinned SHA256 values before extraction or execution. The SDK supplies glslc, which Ubuntu 22.04 does not package.
  • Add 70 to the Linux CUDA architectures.

Both Vulkan assets are required for publication. A Vulkan build failure therefore blocks CPU and Apple releases too. Allowing a release without them would leave affected AMD users on the older fallback after the Studio pin update. CUDA keeps its existing best-effort policy.

After merging, publish the native release, then update Studio's prebuilt pin. unslothai/unsloth#12902 fixes Windows CPU selection when a release contains both CPU and Vulkan assets.

Testing

Live A/B on the AMD DevLab Linux and Windows runners, both with a Radeon 8060S. Old source bfbef5b against current source 7bbffa3, using Q4, 1344x768, 124 frames, two denoising steps and a 20 GiB VRAM budget.

Old Vulkan build Current source
Linux Segfault at a graph-segment transition Video and audio, about 40 minutes
Windows Abort allocating the encoder buffer Video and audio, about 67 minutes

All 124 frames and the audio decoded successfully on both. An isolated backport of the graph-state fix also completed the Linux run.

Linux run / Windows run

  • The hardware runs used binaries from a separate test workflow, with the Linux one built on Ubuntu 24.04. After moving to 22.04, the Linux Vulkan sd-cli and sd-server built, were packaged and extracted through Studio's installer helpers, and loaded in an Ubuntu 22.04 container without the SDK mounted. Their highest required GLIBC symbol version is 2.34. The GPU run was not repeated with this package.
  • This workflow ran end to end on a fork with signing and publication removed (run). All eight build legs and assembly passed. Both Vulkan legs passed the pinned SHA256 checks and found Vulkan 1.4.328 and glslc from the SDK. The fingerprint check covered all eight bundles, and the coverage gate found all seven required assets.
  • The CUDA bundle from that run contains sm_70 kernels in both sd-cli and sd-server, alongside sm_75 through sm_120a.
  • Both SDK downloads matched LunarG's checksum index. Studio selects the new asset names. Actionlint passed.

Not tested:

  • V100 hardware, Windows signing, or publication.
  • GPU runs of the binaries this workflow produces.
  • Visual quality at normal step counts. These were two-step execution checks.
  • The original RX 7900 DeviceLost failure was not reproduced identically on the 8060S.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-06T23:52:45.155956Z 51e2e8e Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@oobabooga

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. What shall we delve into next?

Reviewed commit: 51e2e8e69a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant