Repository navigation
Metal command-buffer limits (50 ops / 50 MB) cost 5–8% decode throughput on launch-heavy models (M3 Ultra measurements) #4521
Description
Device commits a Metal command buffer after max_ops_per_buffer dispatches or when the distinct input buffers referenced exceed max_mb_per_buffer (50 / 50 on Ultra-class devices, backend/metal/device.cpp). For a model whose weight matrices are tens of MB each, the byte rule trips roughly once per layer, so a single-token decode step spans 45–90 command buffers. Each boundary costs about 20 µs of GPU time plus a completion handler on the CPU.
Measured on an M3 Ultra (mlx 0.32.2), GLM-5.3-Flash mixed 4/6-bit (45 layers, MoE + hybrid attention, ~1,600 kernels per token), single-stream decode, same prompts, output hashes compared across arms:
| arm | 20k-token context | short context |
|---|---|---|
| defaults | 35.68 ms/token | 30.91 ms/token |
MLX_MAX_OPS_PER_BUFFER=1000, MLX_MAX_MB_PER_BUFFER=400 |
33.94 (+5%) | 28.71 (+7.7%) |
| defaults again (control) | 35.29 | 30.91 |
Outputs bit-identical, prefill time unchanged, peak memory unchanged during a cold 20k-token prefill. A synthetic chain of 3,200 tiny dependent kernels goes from 8.8 ms to 7.7 ms per step at 200 ops per buffer.
The same variables on a dense Qwen3.8-27B 8-bit model (about 1,000 large matvecs per step) change nothing (52.8 → 53.0 ms/token short, 71.3 → 72.8 at 20k), so the effect is specific to launch-heavy graphs.
Suggestion: scale the byte limit with the size of the largest input buffers, or raise the Ultra defaults; alternatively document the two variables, which are currently undocumented.
A write-up with the per-segment decode profile behind these numbers follows on my blog; happy to share the profiling driver (flag-gated, repeats a segment on sibling layers via mx.depends) if useful.
Activity
zcbenz commentedon Oct 2, 2026
They are documented in https://ml-explore.github.io/mlx/build/html/usage/environment_variables.html and #4562 would make it easier to tune different parts of graph.
I wonder if we should require a minimum number of ops when respecting
MLX_MAX_MB_PER_BUFFERthough, i.e. using something like(buffer_ops_ > 10 && buffer_sizes_ > max_mb)instead of mere(buffer_sizes_ > max_mb). TheMLX_MAX_MB_PER_BUFFERdoes not seem to be good way to tune for large models, and it is not even counting against actual bytes (#3192) .Vissirexa commentedon Oct 18, 2026
Another data point from an M5 Max (40-core GPU, 128 GB, MLX 0.32.4, mlx-lm 0.32.2; defaults are also 50 ops / 50 MB here). Single-stream greedy decode, 512-token prompts, one process per setting. Generated tokens and sampled logits matched across settings in every case.
Changing only
MLX_MAX_MB_PER_BUFFERto 2048 (ops left at 50), code / prose prompts:(The dense changes are relative to default runs before and after the 2048 run. Dense rows replay a saved 512-token prefix, so prefill ran at default settings for those.)
This matches the pattern in the original report: dense models barely move, while the launch-heavy MoE gains the most. On the MoE, custom fused expert kernels gave about +13% at default limits but only about 1–2% on top of the 2048 MB setting. So the two seem to remove the same overhead.
The cost is in prefill. For the MoE, the 512-token prefill peak rose about 0.57 GB at 2048 MB. For the dense 27–31B models, even 512 MB raised the 512-token prefill peak by 1.3–2.1 GB. That supports @zcbenz's idea of requiring a minimum op count before the byte rule triggers, or a decode-only setting, over simply raising the byte default.