Skip to content

Add "config" module - #4647

Open
zcbenz wants to merge 1 commit into
ml-explore:mainfrom
zcbenz:config
Open

zcbenz wants to merge 1 commit into
ml-explore:mainfrom
zcbenz:config

Conversation

@zcbenz

@zcbenz zcbenz commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

Inspired by jax.config, add a new config module that allows overriding configuration options at run time, as a more relaxable alternative to environment variables.

An example is to disable TF32 for certain ops instead of the whole program:

mx.config.update("MLX_ENABLE_TF32", 0)
high_precise_c = a @ b
mx.config.update("MLX_ENABLE_TF32", 1)

There is also a syntax sugar:

with mx.config.scoped_update(MLX_ENABLE_TF32=0):
  high_precise_c = a @ b

Another example is #4562, in which users want to tune MLX_MAX_MB_PER_BUFFER only for a partial graph instead of the whole program.

Closes #4521.
Closes #4595.

This PR also removes env::get_var and replaces all calls with config::get, which is aware of the configs set at run time.

AI usage disclosure: C++ tests

@aidiffuser

Copy link
Copy Markdown

Tested this as the replacement for #4562 (command-buffer limits raised around decode steps only), on one M3 Ultra (512 GB) with stock mlx-lm main.

This PR (7596fca) against its base (9702959). Greedy decoding, 256 tokens, the default 2048-token prefill chunks. Decode tok/s at short / 32k context; percentages are against the mean of four defaults runs (two per build). Generated tokens were identical on every run.

MiMo-V2.6-Flash GLM-5.3-Flash 6-bit
base, defaults 46.2 / 41.7 22.3 / 19.9
this PR, defaults 46.4 / 41.6 22.4 / 19.9
scoped_update(MLX_MAX_OPS_PER_BUFFER=150, MLX_MAX_MB_PER_BUFFER=5000) around decode +5.6% / −1.6% +4.9% / +7.9%
update(...) to 150 / 5000 before decode, restored after +6.2% / −1.4% +4.9% / +7.9%
scoped_update(...) with 50 / 5000 around decode +6.2% / +5.9% +4.1% / +6.5%
  • Reading the limits through config::get on every needs_commit costs nothing measurable: this PR with defaults is within ±0.4% of its base.
  • scoped_update and update give the same result, and peak memory during a 32k prefill is unchanged in every run, since the limits are back to their defaults before prefill.
  • One small thing in the description: scoped_update("MLX_ENABLE_TF32"=0) isn't valid Python. The binding takes keywords, so it's scoped_update(MLX_ENABLE_TF32=0).

Thanks for this, it covers what #4562 was for.

Co-authored with Claude Opus 5.5

@zcbenz

zcbenz commented Oct 9, 2026

Copy link
Copy Markdown
Member Author

Thanks for verifying the change and for catching the python error!

eejd added a commit to eejd/cpp-mlx that referenced this pull request Oct 10, 2026
…_SDPA=0

Review fixes on the ml-explore#4647 backport (both also present in the upstream PR):
- ScopedUpdate had a user-defined copy constructor but the implicit copy
  assignment, which would leave two guards sharing one scope entry and
  erase it twice.
- supports_sdpa_cudnn read MLX_CUDA_USE_CUDNN_SDPA but no longer returned
  early when it is 0.
eejd added a commit to eejd/py-vllm-mlx that referenced this pull request Oct 10, 2026
On GPUs with matrix units (Apple M5+) MLX runs float32 matmul, attention
and convolution at reduced TF32-class precision unless MLX_ENABLE_TF32=0.
The flag selects full float32 (highest) or allows reduced precision
(high, MLX's default), using the names of
torch.set_float32_matmul_precision and JAX default_matmul_precision.

Uses the runtime mx.config module (ml-explore/mlx#4647) when present and
falls back to MLX_ENABLE_TF32 before model load otherwise. Unset leaves
MLX's default and the environment variable in effect.
eejd pushed a commit to eejd/cpp-mlx that referenced this pull request Oct 16, 2026
Adds mx.config / mlx::core::config so MLX_* settings (MLX_ENABLE_TF32 and
the others) can be changed while a process runs, for the process or for a
block of code on the current thread.

Rebased from the v0.32.3 backport (eejd/cpp-mlx macports/v0.32.3 @
68d5378) onto v0.32.4: the gated-delta chunk/checkpoint helpers and the
SDPA VJP / residency / Winograd settings added since v0.32.3 are converted
from env::get_var to config::get as well, matching the PR on main.
Also converts test_fast_gated_delta.py os.environ mutations to
mx.config.update: config::get caches the first env read, so later
os.environ writes in the same process are ignored (also present in the
upstream PR head).

Source: ml-explore#4647 (open; head 0569a18)
Source: 68d537842
Co-authored-by: Cheng <git@zcbenz.com>
eejd added a commit to eejd/cpp-mlx that referenced this pull request Oct 16, 2026
…_SDPA=0

Review fixes on the ml-explore#4647 backport (both also present in the upstream PR):
- ScopedUpdate had a user-defined copy constructor but the implicit copy
  assignment, which would leave two guards sharing one scope entry and
  erase it twice.
- supports_sdpa_cudnn read MLX_CUDA_USE_CUDNN_SDPA but no longer returned
  early when it is 0.
eejd added a commit to eejd/cpp-mlx that referenced this pull request Oct 16, 2026
…he environment

config::get() stored the value it returned, including the default passed by
the first caller. Defaults that depend on the call site were then pinned for
the whole process: gated_delta_chunk_size(T) passes T > 8 ? (nax ? 16 : 8) : 1,
so a server that decoded (T=1) before it prefilled got chunk size 1 for every
later prefill (and the reverse order pinned 8/16 for decode).

A variable that is set in the environment is still read once and cached
(first-read semantics of ml-explore#4647 are kept); an unset variable is remembered as
unset and the caller's default is returned as is. update() and scoped_update()
are unchanged.

Divergence from upstream ml-explore#4647, whose config.cpp has the same
pattern (see UPSTREAM.md). C++ and Python tests: get(name, 1), then get(name, 8)
must be 1 then 8 (fail before, pass after).
eejd added a commit to eejd/cpp-mlx that referenced this pull request Oct 16, 2026
…x.config

test_conv (MLX_CONV_WINOGRAD) and test_fast_gated_delta (GATED_DELTA_VJP_FALLBACK,
GATED_DELTA_CHUNK_VJP) changed the process environment, which config::get reads
only once and caches, so after the backport the Winograd-off and VJP
fallback/chunk cases silently ran the default path. Converted as upstream
ml-explore#4647 does (these lines landed in v0.32.4, after the PR base).
eejd added a commit to eejd/cpp-mlx that referenced this pull request Oct 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants