Repository navigation
Add "config" module - #4647
Open
zcbenz wants to merge 1 commit into
Open
Add "config" module#4647zcbenz wants to merge 1 commit into
zcbenz wants to merge 1 commit into
Conversation
This was referenced Oct 8, 2026
|
Tested this as the replacement for #4562 (command-buffer limits raised around decode steps only), on one M3 Ultra (512 GB) with stock mlx-lm main. This PR (7596fca) against its base (9702959). Greedy decoding, 256 tokens, the default 2048-token prefill chunks. Decode tok/s at short / 32k context; percentages are against the mean of four defaults runs (two per build). Generated tokens were identical on every run.
Thanks for this, it covers what #4562 was for. Co-authored with Claude Opus 5.5 |
Member
Author
|
Thanks for verifying the change and for catching the python error! |
eejd
added a commit
to eejd/cpp-mlx
that referenced
this pull request
Oct 10, 2026
…_SDPA=0 Review fixes on the ml-explore#4647 backport (both also present in the upstream PR): - ScopedUpdate had a user-defined copy constructor but the implicit copy assignment, which would leave two guards sharing one scope entry and erase it twice. - supports_sdpa_cudnn read MLX_CUDA_USE_CUDNN_SDPA but no longer returned early when it is 0.
eejd
added a commit
to eejd/py-vllm-mlx
that referenced
this pull request
Oct 10, 2026
On GPUs with matrix units (Apple M5+) MLX runs float32 matmul, attention and convolution at reduced TF32-class precision unless MLX_ENABLE_TF32=0. The flag selects full float32 (highest) or allows reduced precision (high, MLX's default), using the names of torch.set_float32_matmul_precision and JAX default_matmul_precision. Uses the runtime mx.config module (ml-explore/mlx#4647) when present and falls back to MLX_ENABLE_TF32 before model load otherwise. Unset leaves MLX's default and the environment variable in effect.
eejd
pushed a commit
to eejd/cpp-mlx
that referenced
this pull request
Oct 16, 2026
Adds mx.config / mlx::core::config so MLX_* settings (MLX_ENABLE_TF32 and the others) can be changed while a process runs, for the process or for a block of code on the current thread. Rebased from the v0.32.3 backport (eejd/cpp-mlx macports/v0.32.3 @ 68d5378) onto v0.32.4: the gated-delta chunk/checkpoint helpers and the SDPA VJP / residency / Winograd settings added since v0.32.3 are converted from env::get_var to config::get as well, matching the PR on main. Also converts test_fast_gated_delta.py os.environ mutations to mx.config.update: config::get caches the first env read, so later os.environ writes in the same process are ignored (also present in the upstream PR head). Source: ml-explore#4647 (open; head 0569a18) Source: 68d537842 Co-authored-by: Cheng <git@zcbenz.com>
eejd
added a commit
to eejd/cpp-mlx
that referenced
this pull request
Oct 16, 2026
…_SDPA=0 Review fixes on the ml-explore#4647 backport (both also present in the upstream PR): - ScopedUpdate had a user-defined copy constructor but the implicit copy assignment, which would leave two guards sharing one scope entry and erase it twice. - supports_sdpa_cudnn read MLX_CUDA_USE_CUDNN_SDPA but no longer returned early when it is 0.
eejd
added a commit
to eejd/cpp-mlx
that referenced
this pull request
Oct 16, 2026
…he environment config::get() stored the value it returned, including the default passed by the first caller. Defaults that depend on the call site were then pinned for the whole process: gated_delta_chunk_size(T) passes T > 8 ? (nax ? 16 : 8) : 1, so a server that decoded (T=1) before it prefilled got chunk size 1 for every later prefill (and the reverse order pinned 8/16 for decode). A variable that is set in the environment is still read once and cached (first-read semantics of ml-explore#4647 are kept); an unset variable is remembered as unset and the caller's default is returned as is. update() and scoped_update() are unchanged. Divergence from upstream ml-explore#4647, whose config.cpp has the same pattern (see UPSTREAM.md). C++ and Python tests: get(name, 1), then get(name, 8) must be 1 then 8 (fail before, pass after).
eejd
added a commit
to eejd/cpp-mlx
that referenced
this pull request
Oct 16, 2026
…x.config test_conv (MLX_CONV_WINOGRAD) and test_fast_gated_delta (GATED_DELTA_VJP_FALLBACK, GATED_DELTA_CHUNK_VJP) changed the process environment, which config::get reads only once and caches, so after the backport the Winograd-off and VJP fallback/chunk cases silently ran the default path. Converted as upstream ml-explore#4647 does (these lines landed in v0.32.4, after the PR base).
eejd
added a commit
to eejd/cpp-mlx
that referenced
this pull request
Oct 16, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Inspired by
jax.config, add a newconfigmodule that allows overriding configuration options at run time, as a more relaxable alternative to environment variables.An example is to disable TF32 for certain ops instead of the whole program:
There is also a syntax sugar:
Another example is #4562, in which users want to tune
MLX_MAX_MB_PER_BUFFERonly for a partial graph instead of the whole program.Closes #4521.
Closes #4595.
This PR also removes
env::get_varand replaces all calls withconfig::get, which is aware of the configs set at run time.AI usage disclosure: C++ tests