Repository navigation
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Layer normalization computes variance as
mean(x**2) - mean(x)**2, which loses the variance when features share a large offset. For FP32 input[10000, 10001, 10002, 10003], the implementation returns approximately[-474.34, -158.11, 158.11, 474.34]instead of[-1.34, -0.45, 0.45, 1.34]. Squaring FP16 inputs can also overflow even when the normalized result is representable.Use
torch.var_mean(..., unbiased=False)for population variance and accumulate FP16/BF16 statistics in FP32. Keep FP64 statistics and the existing output and affine type-promotion behavior. The annotated explanation now describes the numerically stable variance calculation.The new tests cover common-offset invariance, large and narrowly spread low-precision values, singleton and multidimensional normalized shapes, affine/output dtypes, input and parameter gradients, and autograd gradcheck against a double-precision
torch.nn.functional.layer_normreference.Validation on an isolated GitHub Actions CPU runner with PyTorch 2.10.0+cpu and Python 3.12.14: the complete affected suite gives 21 passed. Restoring the original implementation gives 14 failed, 7 passed. Both validation steps completed successfully. There are two existing/environment warnings: the annotated docstring's escape sequence and PyTorch's NumPy initialization warning because NumPy was not installed in that isolated job.
Native CPU validation run
The local PyTorch 2.14.1 CPU FP32/FP64/BF16 subset also passes (16 passed, 5 deselected); FP16 arithmetic was validated in the isolated runner.
git diff --checkpasses. No GPU execution, model training or full repository suite was run. This repository has no checked-in test/lint configuration or mandatory contribution template at the audited revision.