Repository navigation
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AdaBelief's positional call to
RAdam.__init__passesamsgradasoptimized_update,degenerate_to_sgdasamsgrad, anddefaultsasdegenerated_to_sgd. The actual optimizer can therefore differ from its requested configuration. With the defaults, a scalar parameter initialized to 1 and gradient 1 remains at 1 for the first five steps withlr=0.1; the requested SGD fallback should update it to 0.9 on the first step. Passing an unrelated nonempty defaults dictionary currently changes whether those SGD steps happen, while its entries never reach the parameter groups.Forward these arguments by name. Keep
optimized_update=Falseexplicitly so the existing default epsilon convention remainssqrt((s+eps)/(1-beta2**t)) + epsand changing AMSGrad no longer changes epsilon placement. Correct the constructor documentation to describe that convention.The tests compare actual updates and optimizer moments with the mathematical recurrence over 16 steps, including the transition from SGD to rectified updates. They cover all combinations of rectification, AMSGrad and SGD fallback, a custom learning-rate default used by a subclass, per-group AMSGrad overrides, and parameters without gradients.
Validation:
python -m pytest tests/test_ada_belief.py -q: 12 passed on CPU with PyTorch 2.14.1.git diff --check: passed.There are 13 existing documentation escape warnings. No GPU execution, model training, full repository suite or execution of the official AdaBelief implementation was performed. That implementation was inspected as an additional reference for independent AMSGrad and SGD-fallback settings and the bias-corrected nonrectified denominator.