Skip to content

fix: honor AdaBelief AMSGrad and SGD fallback settings - #320

Open
Afloat16 wants to merge 1 commit into
labmlai:masterfrom
Afloat16:fix/ada-belief-configuration
Open

Afloat16 wants to merge 1 commit into
labmlai:masterfrom
Afloat16:fix/ada-belief-configuration

Conversation

@Afloat16

@Afloat16 Afloat16 commented Oct 2, 2026

Copy link
Copy Markdown

AdaBelief's positional call to RAdam.__init__ passes amsgrad as optimized_update, degenerate_to_sgd as amsgrad, and defaults as degenerated_to_sgd. The actual optimizer can therefore differ from its requested configuration. With the defaults, a scalar parameter initialized to 1 and gradient 1 remains at 1 for the first five steps with lr=0.1; the requested SGD fallback should update it to 0.9 on the first step. Passing an unrelated nonempty defaults dictionary currently changes whether those SGD steps happen, while its entries never reach the parameter groups.

Forward these arguments by name. Keep optimized_update=False explicitly so the existing default epsilon convention remains sqrt((s+eps)/(1-beta2**t)) + eps and changing AMSGrad no longer changes epsilon placement. Correct the constructor documentation to describe that convention.

The tests compare actual updates and optimizer moments with the mathematical recurrence over 16 steps, including the transition from SGD to rectified updates. They cover all combinations of rectification, AMSGrad and SGD fallback, a custom learning-rate default used by a subclass, per-group AMSGrad overrides, and parameters without gradients.

Validation:

  • python -m pytest tests/test_ada_belief.py -q: 12 passed on CPU with PyTorch 2.14.1.
  • Restoring the original implementation: 8 failed, 4 passed.
  • git diff --check: passed.

There are 13 existing documentation escape warnings. No GPU execution, model training, full repository suite or execution of the official AdaBelief implementation was performed. That implementation was inspected as an additional reference for independent AMSGrad and SGD-fallback settings and the bias-corrected nonrectified denominator.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant