You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Adds the com.microsoft::RMSNormRotary (version 1) CUDA contrib operator for Qwen3.8-Flash-Next QSA query/key normalization and partial rotary embedding:
The new exact CPU path computes this RMS from sum_sq += value * value at line 107. On builds that permit fused multiply-add contraction, that expression removes the documented FP32 square-rounding boundary. For FP32 input [1.58784556, 2.48118043, 1.39505327], separate rounding sums to 10.623682975769043, while contracted accumulation gives 10.62368392944336 and changes normalized outputs. Force the square to round before accumulation in exact mode, preserve the legacy expression otherwise, and add an FMA-sensitive FP32 regression.
A schema-valid node with outputs ["Y", ""] omits Y_fp32, but getNumOutputs() still returns 2. With the default arithmetic mode, this block rejects the model during Graph::Resolve even though no widened output was requested. Gate it with ctx.hasOutput(1), which checks whether the output edge exists, and add a regression using AddOptionalOutputEdge<float>().
These exact-mode tests cover only FP16/BF16 storage. For their bounded values, multiplication is exact in FP32, so they cannot detect accidental fused multiply-add contraction. Add an FP32 cancellation test with independently specified, bitwise-checked expectations: (1 + 2^-23) * (1 - 2^-23) - 1 rounds to zero with separate operations but to -2^-46 when contracted. Reusing the version-1 float kernel as the reference would not independently verify this boundary.
Pushed 5c02717: dedicated CUDA BF16 RMSNormRotary operator, schema/registrations, exact-parity and contract tests, and optional prefill numerical controls. Preserved the reviewer-added deterministic-compute argument separately from reciprocal-multiply mode. New native translation units, CPU/CUDA paths, schema and tests compile using the existing build configuration; WebGPU is unavailable here and was not compiled. The previously tested combined runtime passed 125 focused native tests (3 WebGPU skips) and 33 optimizer tests, with exact 512-token parity on 44 requests. Those combined-runtime results are not an independent build/link qualification of this PR branch. Paired decode improvement was +0.81% at 2K and +0.77% at 8K; 994 to 946 kernels/cycle. Full accuracy evaluation is starting separately.
…2 QMoE, and PR cleanups
- Extend FoldMid2/FoldTail to head_rows<=8 and pack tokens[4..7] into ring record words 1, 2, 3, and 5.
- Add state_update_capacity (0..8) support to CausalConvWithState and enable state_update_capacity > 0 in QwenS2 GatedDeltaNetDecodeWarpKernel.
- Raise kMaxProfiledExpandedRows to 80 in moe_gemv.h and kFusedRouteMaxRows to 8 in qmoe_fused_route.cuh; add fast INT3 and offline SM80 INT2/INT4 prepack QMoE support.
- Eliminate pq[8][ROWS] local register array in HcUpPreKernel and sync PR cleanups (#33469, #33470, #33471, #33472, #33474, #33548, #33569).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Tianlei Wu (tianleiwu)
changed the title
Summary
Adds the
com.microsoft::RMSNormRotary(version 1) CUDA contrib operator for Qwen3.8-Flash-Next QSA query/key normalization and partial rotary embedding:com.microsoft::RMSNormRotary(onnxruntime/contrib_ops/cuda/bert/rms_norm_rotary.{h,cc,cu}):[B, S, heads, stride]slice[input_start : input_start + input_width]) with non-interleaved partial RoPE (rotary_embedding_dim) and flattened[B, S, heads * width]output in a single 64-thread 2-warp shuffle kernel (RMSNormRotaryKernel).use_fp32_arithmetic=1) and prefill arithmetic.RotaryEmbedding(ver 2)andBranchwiseRMSNorm(ver 2)changes (BranchwiseRMSNorm(ver 1)onmainalready supportsBFloat16scales).