Skip to content

Apply decoupled weight decay at the current scheduled learning rate - #322

Open
Afloat16 wants to merge 1 commit into
labmlai:masterfrom
Afloat16:fix-scheduled-decoupled-weight-decay
Open

Afloat16 wants to merge 1 commit into
labmlai:masterfrom
Afloat16:fix-scheduled-decoupled-weight-decay

Conversation

@Afloat16

@Afloat16 Afloat16 commented Oct 6, 2026

Copy link
Copy Markdown

Noam, warmup and cosine schedules change the actual adaptive update rate while decoupled weight decay still uses the group maximum learning rate. Pass the current step rate only to decay, preserving group schedule parameters and the existing three-argument decay extension API.

Independent Adam recurrences check moments and parameter trajectories across schedules, both update formulas, coupled/absolute controls, and dynamic rate hooks with parameter groups and missing gradients. All four optimizer overrides are fixed as one scheduling defect.

Validation: the same 32-case regression file gives 14 failures and 18 passes on upstream, then 32 passes with this change. Tested with Python 3.12.10, PyTorch 2.14.1 CPU, pytest 9.1.1 and official labml 0.4.168. No skips or collection errors. The recorded upstream tree has no formal original unit suite; GPU and full-model training are untested.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant