Repository navigation
Add exact normalization and rotary CUDA operator modes - #33459
Closed
Akshay Sonawane (apsonawane) wants to merge 87 commits into
Closed
Akshay Sonawane (apsonawane) wants to merge 87 commits into
Akshay Sonawane (apsonawane) wants to merge 87 commits into
Conversation
## Summary Follow-up to the review of #33110. The shared WebGPU data-transfer wrapper does not need to own a context. A session stream already supplies its owning execution provider's transfer state, while a streamless copy can select its registered context from the GPU tensor's memory-device ID. - Remove the cached context, initialization mutex, and context-ID argument from the internal transfer creator. - Route session copies directly through their stream. Retain the tensor-selected registered context only for each streamless copy, including context zero, and flush successful streamless GPU-to-GPU copies. - Document that memory-device identity does not register Dawn objects, the current Env/global allocator context-zero restriction, and the registration requirements for external-context copies. Allocator/session context IDs and the public C API are unchanged. This does not add pre-session external-context registration. ## Validation - Scoped C++ lintrunner and staged whitespace checks passed. - Fresh Windows x64 Release external-Dawn plugin build on upstream main `864d87126864c64651f58a9b51d9006ad71b2c78` passed. - All five native-host modes passed against the fresh ORT/plugin DLLs: baseline, host-device, missing-sync, callback-failure, and missing-proc-table. The host executable was reused after verifying its source was unchanged. - Host-device coverage includes coexisting default/external context allocation and transfers, shader-cache reuse/isolation, and device usability after ORT teardown. Negative modes rejected invalid inputs or contained callback failures as expected. - The same follow-up also passed the five host modes in both built-in and plugin configurations before rebasing onto current main.
### Description CPU `QuantizeLinear` accepts independent input and scale types in opset 23, but reads the scale using the input type and ignores `precision`. Read each tensor with its own type and perform division in FLOAT or FLOAT16, using the scale type when `precision` is unspecified. Keep the existing FP32 computation and pre-opset-23 paths. ### Motivation and Context Part of #32719. For example, FP16 `15.5` with FP32 scale `1.0003` should quantize to `15`; rounding the scale to FP16 first gives `16`. The regressions cover this distinction, explicit precision, quantization granularities, saturation, and packed tails. This covers CPU quantization. CPU dequantization is handled separately in #32728; CUDA remains separate work. Implemented with Codex assistance. --------- Signed-off-by: TANGBUDU <tangbudu@gmail.com>
### Description <!-- Describe your changes. --> - Preserve compile-output checks while skipping file-backed runtime cases unsupported by the WASM test module. - Exclude the host-filesystem external-initializer test on Emscripten; retain native coverage. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to it here. --> The `wasm_Debug / build-wasm` job failed because these tests attempted to load external data without a `Module.MountedFiles` mapping. WASM’s test module does not provide that mapping. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>
## Summary - Replace the single-token QMoE gate full expert bubble sort with a stable parallel TopK reduction for normalized routing. Preserve descending-logit/index-ascending ties, softmax weight order, and the existing full sort when routing weights are not normalized (the denominator then needs all experts). - Cover boundary ties in 3-, 5-, 6-, and 512-expert pools with nonzero expert outputs. Skip the 512-expert case when either adapter workgroup limit is below 512. - Assert the selected gate indices directly for FP16/FP32 negative-infinity logits (`[0, -inf, -inf]` selects `[0, 1]`) and for all-equal logits with `k == num_experts` (`[0, 1, 2, 3]`). The negative-infinity assertion fails with the original comparator (it selects the invalid sentinel index 3). - Exercise FP16/FP32 nonnormalized routing with an excluded expert contributing appreciably to the all-expert softmax denominator. ## Validation - On head `45037061757c`, merged `origin/main` at `9f6baac7319a` (including #33168) without rebasing or gate conflicts. Rebuilt both the static test-all and split provider-test targets; all 28 focused WebGPU MoE tests passed on H200, including FP16/FP32 gate-index readback. Checked free GPU memory before the run; fresh CI for this head is in progress. - Main-based static WebGPU build and all 22 focused WebGPU QMoE tests passed on H200 after merging `main` at `aac791ef2408a266d8ee59dd5fa0ce11d1a5a0bb`, including the new tie, gate-index, and nonnormalized-routing regressions. - Qwen3.8-Flash-Next INT4 RTN/INT8-head export, 512 experts, k=10, FP16 I/O, one-token chunks, H200: 16-token prefill 11.64 s -> 0.90 s and steady decode ~0.674 s -> ~0.045 s/token. Baseline versus optimized logits were bit-identical (248,320 logits), and the first five generated token IDs matched. This full-model comparison used a QSA-enabled experimental build carrying the same single-token gate change because the export also requires the not-yet-merged SparseAttentionIndexer kernel; the isolated main-based plugin cannot load that export today. The PR itself does not depend on the QSA changes. - This PR changes only the single-token gate. Separate GPU-only multi-token prefill experiments are intentionally excluded pending model-quality validation. ## Main updates - October 5: merged `main` and added the follow-up requested on merged #32878: two experts use distinct INT2/INT4 packed weights as well as scales, with matching FP16-aware numerical references. The fixture explicitly rejects unsupported 8-bit encodings. - October 6: merged `main` at `aac791ef2408a266d8ee59dd5fa0ce11d1a5a0bb`; added direct gate-index assertions, 5-/6-expert pool tests, adapter-limit skip, and the full-denominator regression. That head was superseded by the fixture alignment fix below. - October 6 follow-up: pad the direct gate-index test fixture buffers to 4-byte boundaries for odd FP16 expert counts. On the prior head, macOS Dawn rejected a six-byte storage buffer and an eight-byte staging copy. The prior-head Windows job reported a lost device at the same gate-index test (0 ms), followed by cascading failures; its log does not establish whether the fixture caused the device loss. - October 7 follow-up: current-head arm64 Debug reproduced a real test-fixture readback error: `BufferManager::Download` rounds the copy to 16 bytes, but the `k=2` indices buffer was only 8 bytes. Align that buffer to 16 bytes. Windows novcpkg lost the WebGPU device at the same gate-index test and subsequently failed 1,538 unrelated tests; the static vcpkg aggregate reported the dependent test failure despite successful compilation. All 28 focused WebGPU MoE tests pass after the fix on H200. - October 7 main refresh: merged `main` at `9f6baac7319a` (including #33168), rebuilt the split static WebGPU provider-test target, and passed all 28 focused H200 WebGPU MoE tests on head `45037061757c`. Prior-head CI failures do not describe this new head; current-head checks are running. ## October 8 main refresh Merged `origin/main` at `864d87126864c64651f58a9b51d9006ad71b2c78` (including #33203) into the gate branch, resolving the test include conflict by retaining both sets of includes. Updated the direct gate-buffer test to call upstream `WebGpuDevice(0)` after the context-ID API change. On head `813f7999e55de5a8d29a7a02dfc869d514f327b4`, the static WebGPU provider-test build succeeded and all 34 focused H200/Vulkan `MoETest.QMoETest_WebGPU*` cases passed. Fresh CI is queued; no new-head CI result is claimed yet. ## Matched synthetic whole-inference benchmark (October 8) H200/Vulkan ONNX Runtime `perf_test`, synthetic single-token block-FP8 QMoE with 256 experts and top-4 routing, three 100-run trial averages: merged #32794 baseline **7.09 ms/run** median-trial mean versus the integrated gate branch **5.56 ms/run** (~**1.28x**). Individual trial averages ranged **5.71–7.12 ms** for baseline and **5.46–5.89 ms** for the integrated branch; these overlapping ranges warrant a variance caveat. This measures whole inference, **not** the isolated gate kernel or a full checkpoint. The separate combined branch passed **52/52** focused tests; that result is not claimed as an isolated #33058-head test run. ## October 8 documentation follow-up Added [WebGPU QMoE single-token gate documentation](https://github.com/sayanshaw24/onnxruntime/blob/8d38e0e31be0216e7d3b937578efa113fccee4ca/docs/contrib_ops/webgpu/qmoe_gate.md) on head `8d38e0e31b`, covering normalized top-k, tie and -inf handling, nonnormalized fallback, adapter limits, and benchmark scope. The previous-head Windows documentation-build job stopped while compiling unrelated CUDA sources before document generation, so its dependent validation failure does not establish a generated-doc mismatch. The previous-head Android job failed during NDK setup with a corrupt downloaded archive. Fresh-head CI is running. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
### Description This PR allows SM120 GEMM configurations on SM121 devices in the QMoE kernel's dispatch. ### Motivation and Context The dispatch check previously required an exact match between the configuration and device SM versions, rejecting SM120 configurations on SM121 devices. This change permits that pairing while retaining the check for other mismatches. This PR also fixes an integration issue for Surface Laptop Ultra devices that contain an RTX Spark GPU.
### Description
Large ONNX models often store weights in external files. The existing
native WebGPU loading path materializes each weight in tensor-sized,
CPU-accessible memory before uploading it to the GPU. For large tensors
and multi-gigabyte models, these allocations increase CPU memory
pressure, address-space activity, and session creation time.
This change adds an accelerated Windows loading path that streams
external weights through a bounded pool of twelve reusable 16 MiB
Direct3D 12 (D3D12) upload buffers. File reads and GPU copies operate in
chunks, so staging memory remains bounded instead of scaling with
individual tensor sizes. The loader copies each chunk into its final
GPU-local tensor resource, which WebGPU then imports into Dawn.
```text
External weight file
-> mapped D3D12 upload buffer
-> D3D12 copy queue
-> GPU-local tensor resource
-> Dawn shared-buffer import
```
File reads, GPU copies, and upload-buffer reuse overlap through Windows
events and D3D12 fences. Weight payloads are not materialized in
application CPU memory, and the application does not perform an
additional CPU payload copy.
### Design
The reusable loader,
`onnxruntime::windows::d3d12::D3D12FileBufferLoader`, is part of the
shared Windows platform layer. It does not depend on Dawn, WebGPU, ONNX
Runtime tensors, or execution-provider interfaces.
The WebGPU integration:
- Selects the requested physical adapter through DirectX Graphics
Infrastructure (DXGI).
- Pins Dawn and the loading device to the same adapter by locally unique
identifier (LUID).
- Prepares external tensor ranges and invokes the shared loader.
- Imports completed D3D12 resources into Dawn.
- Retains the D3D12 resources and heaps for the required lifetime.
The shared loader provides:
- Overlapped, unbuffered Win32 file reads.
- Twelve persistent 16 MiB mapped upload buffers.
- Event-driven read completion and D3D12 copy-queue uploads.
- Overlap between destination allocation and the initial read wave.
- Placed resources packed into heaps targeted at no more than 512 MiB.
- Committed resources for tensors that are too large for those heaps.
- Support for multiple external-data files and non-contiguous ranges.
- GPU-side handling of aligned tensor padding.
- Bounds validation, cancellation, error propagation, and explicit
resource ownership.
### Provider option
Enable the feature with:
```json
{
"weightLoadAcceleration": "preferred"
}
```
Supported values:
- `off`: Use the existing initializer loading path.
- `preferred`: Use accelerated loading when available; otherwise, fall
back to the existing path.
- `required`: Require accelerated loading and return the underlying
error if initialization fails.
Cancellation is always propagated and never triggers fallback.
### Performance
Measurements used a decoder-only model with:
- 454 external tensors.
- 1,291,602,752 bytes of weight data.
- 452 of 453 adjacent tensor boundaries contiguous.
- 3,072 non-tensor bytes within the covered file range.
Test system:
- Intel Core Ultra 9 285K.
- Intel Graphics (`0x8086:0x7D67`).
- NVIDIA GeForce RTX 4070 (`0x10DE:0x2709`).
- SK hynix PC801 NVMe solid-state drive.
#### ONNX Runtime session creation
Nine fresh `onnxruntime_perf_test` processes were run for each mode and
GPU preference.
| GPU preference | GPU | `off` median | `preferred` median | Improvement
|
|---|---|---:|---:|---:|
| High performance | NVIDIA GeForce RTX 4070 | 1.562 s | 0.715 s | **846
ms / 54.2%** |
| Low power | Intel Graphics | 1.420 s | 0.669 s | **751 ms / 52.9%** |
Observed ranges, including the first process:
| GPU | `off` range | `preferred` range |
|---|---:|---:|
| NVIDIA | 1.534-2.826 s | 0.665-0.840 s |
| Intel | 1.400-2.430 s | 0.605-0.783 s |
The first `off` process on each GPU was a cold-start outlier. The
`required` mode uses the same loading path as `preferred`; only its
fallback policy differs.
#### Standalone comparison with DirectStorage
A 15-repetition comparison alternated the path order, excluded the first
run of each path from the steady-state median, and verified every
destination byte.
| GPU | DirectStorage | This implementation | Difference |
|---|---:|---:|---:|
| NVIDIA | 225.097 ms | 187.070 ms | **38.028 ms / 16.9% faster** |
| Intel | 262.044 ms | 241.538 ms | **20.506 ms / 7.8% faster** |
Diagnostics indicate that the new path is storage-bound:
| GPU | Raw unbuffered read | GPU copy only | Approximate rates |
|---|---:|---:|---|
| NVIDIA | 182.050 ms | 52.087 ms | 7.10 GB/s read, 24.80 GB/s copy |
| Intel | 210.477 ms | 68.945 ms | 6.14 GB/s read, 18.73 GB/s copy |
Byte-for-byte verification passed for all 1,291,602,752 payload bytes on
both loading paths and both GPUs.
### Validation
- Static, shared, and external Dawn provider and test targets build
successfully.
- External-data-loader lifecycle and manager tests pass.
- WebGPU context and provider tests pass with static and shared Dawn
configurations.
- Device-precondition tests skip where expected.
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
## Description Fixes #32858. `PadFusion` can replace explicit zero padding with `MaxPool`'s implicit padding. These are not equivalent for inputs that can contain negative values: explicit zeros participate in the maximum, while implicit padding is excluded. For example, with input `[-1, -2]` and a leading zero pad, the first pooled value should be `0`, but the fused graph can return `-1`. ## Changes - Restrict `Pad` -> `MaxPool` fusion to `UINT8`, where zero is the minimum representable value. This also applies when a `Cast` is present before `MaxPool`. - Skip Pool fusion when the merged padding would be invalid (`pad >= kernel_shape`). - Avoid fusion when `ceil_mode` and explicit trailing padding could change the output shape. - Handle missing shape information when removing `Pad` before `Cast`. - Add regression tests for supported and unsupported types, Cast paths, padding boundaries, and `ceil_mode`. Existing float MaxPool model tests now verify that fusion is skipped.
### Description Normalize negative reduction axes before constructing CPU output shapes for empty tensors. Add regression coverage for `ReduceSum`, `ReduceProd`, and `ReduceMean` with axes supplied as an input, both `keepdims` settings, and the legacy attribute-based axes path. Tests: - `cmake --build build/Linux/Release --target onnxruntime_provider_test -j 8` — passed - `cd build/Linux/Release && ./onnxruntime_provider_test --gtest_filter=ReductionOpTest.EmptySetNegativeAxis` — passed (1 test) - `cd build/Linux/Release && ./onnxruntime_provider_test --gtest_filter='ReductionOpTest.*EmptySet*:ReductionOpTest.empty_set_*'` — passed (28 tests) ### Motivation and Context The empty-input reduction path compared raw negative axes with non-negative dimension indices, so an axis such as `-1` was never reduced and produced an incorrect output shape. The normal non-empty path already normalizes axes. Fixes #33230. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
### Description - Add FP16/BF16 H512 XQA decode kernels with FP32 online softmax and partial-result merging. - Support BNSH/BSNH caches, integral positive GQA ratios, local windows, and attention sinks. - Align runtime scratch geometry, workspace recipes, bounds, estimation, and eligibility checks. - Prefer native grouped H512 attention where eligible instead of oversized expanded-KV fallback workspaces. - Add documented kernel, eligibility, workspace, and cache-immutability regressions; retain existing smaller-head dispatch paths. - Preserve main's shared route-eligibility helpers and integrate H512 geometry into the shared XQA predicate. ### Motivation and Context Gemma 4 grouped attention includes 512-wide heads. This adds a bounded decode implementation and consistent workspace accounting rather than expanding the cache to query-head count. ### Validation After merging main at `05517e129656033b13a1617f1d94d0bdc9f0dff1`, resolved commit `e4dedff734cadcc2df828f176aaecf2a54ce3312` was rebuilt in an isolated Release CPU/CUDA build on Linux/A100 (SM80), CUDA 13, QUICK_BUILD off. All 90 native GQA operator tests and 106 focused internal CUDA kernel/workspace/eligibility tests passed. The 13 operator skips are 12 unavailable WebGPU tests and one FP8 test requiring SM89+. Both direct FP16/BF16 H512 kernel tests and all four typed instances of the new unquantized-H512 geometry regression ran and passed. All 17 PR files pass configured lint and whitespace checks. No full-suite, Windows, ROCm, or other GPU-architecture validation. This is one of four independent ORT drafts. Prefer merging the companion GQA cache/window draft #33175 first so the native fallback also receives its window-bound correction; this branch was built and tested without that draft applied. ### Conflict Resolution Merged main without rewriting the original tested commit. The resolution retains main's XQA and MEA helper refactors, adds H512 geometry to the shared XQA predicate, keeps the native-H512 MEA exclusion, and preserves both operator tests that collided at the same insertion point. No other companion draft commits were applied. ### Latest Main Merge (2026-10-09) Resolved the current conflict by merging main at `e379df24dbe4b0c6912282f30fe3f5ae4c1836d4` in `d73ed2d1538ed3dc02a2762057156c122eb6eb00` without rewriting branch history. The operator-test conflict was two independent insertions at the same location: both the native H512 fallback eligibility regression and the upstream FlashAttention L2-cache block-size regressions are preserved, with their original platform guards. Upstream GQA runtime changes merged automatically. For this merge, rebuilt the resolved GQA test translation unit against the merged H512 headers and ran all three conflict-local header-only helper tests successfully. The test executable reused retained backend libraries; this is not a fresh H512 numerical-kernel validation. The automatically merged CUDA GQA runtime source passed a compile check. All 17 remaining PR files passed configured lint and whitespace checks, and editor diagnostics reported no errors in the resolved file. The full native H512 runtime results above describe the earlier validated revision; those numerical tests and a full native rebuild were not rerun for this merge. CI for the new merge commit remains pending.
# 🚀 WebGPU FP16 MatMul Accuracy Fix ## 🐛 Bug The Qwen 2.5 0.5B GenAI WebGPU test produced logits that disagreed with the CPU reference. The issue was isolated to the generic packed FP16 MatMul kernel with shape `M=3, K=896, N=4864`. The shader stored its accumulator as `vec4<f16>`. Each addition was therefore rounded to FP16, causing significant error over the long `K=896` reduction. The worst single-node WebGPU-to-FP32 error was `0.7760754`. ## 🔧 Fix Packed MatMul now supports FP32 accumulation. FP16 inputs and workgroup storage remain unchanged, cached operands are widened to FP32 for multiplication and accumulation, and the result is converted back to the output type when written. ## 🎯 Scope This PR fixes FP16 accumulation in the generic packed ONNX `MatMul` implementation, covering both vec4 and scalar paths. FP16 atomic Split-K is disabled for this implementation to avoid rounding or overflowing partial sums before the final reduction. ONNX `Gemm` is outside this PR's scope. Its packed caller retains the default `use_f32_accumulation = false`, so FP16 `Gemm` accumulation is unchanged. The shared helpers remain opt-in; this PR does not enable FP32 accumulation for every caller or change vendor-specific MatMul kernels. ## ✅ Validation 1. **🧪 ONNX Runtime unit test** Added `MathOpTest.MatMulPackedFp16LongReductionUsesFloat32Accumulator`. Before the fix, it failed with expected `8.953125`, actual `9.140625`, and error `0.1875`. It passes after the fix. 2. **🤖 GenAI regression test** Ran `test_webgpu_paged_export_runs_prefill_and_decode`, the Qwen WebGPU test that exposed the issue. It passed with the rebuilt plugin package. The standalone full-model repro also passed with matching argmax (`4`) and a maximum logit difference of `0.101529`. --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Prathik Rao <prathikrao@microsoft.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
### Description <!-- Describe your changes. --> Harden model-package EP recreation, path handling, and public output contracts. - Preserve selected EP factories, device defaults, custom domains, and caller-supplied provider chains before model load. - Reject silent loss of the selected EP and keep query storage lifetimes stable. - Use UTF-8 consistently across the standalone package library and ORT integration. - Define the standalone C API contract that failed calls leave output parameters unchanged, with sentinel regressions. - Remove the unused, unreleased `ModelPackage_Validate` authoring API and its dedicated report surface. - Adapt the hardening work to the stable `OrtModelPackageApi` now present on `main`. Validation: - `lintrunner` on changed model-package C/C++ files — passed. - Standalone `model_package` Release build with `MODEL_PACKAGE_BUILD_TESTS=ON` — passed. - `ctest --test-dir <standalone-build> -C Release --output-on-failure` — 4/4 suites passed. - `cmake --build build\WindowsVS18\RelWithDebInfo --config RelWithDebInfo --target onnxruntime_autoep_test --parallel 4` — passed. - `onnxruntime_autoep_test.exe --gtest_filter="ModelPackageApiTest.*:ModelPackageTest.*"` — 27/27 tests passed. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Model-package variant selection and session creation must preserve the selected EP configuration and expose predictable public C API behavior. The standalone validation API had no runtime, GenAI, repository-production, or indexed external consumers and had not shipped in a release, so it is removed before becoming a compatibility burden.
### Description - Add profiler-selected symmetric INT4 group32 M=1 CUDA decode geometries: column tiles 2/4/8 with 128/256 threads. Keep the reviewed default dispatcher and CUTLASS candidates as fallbacks; select the measured tactic rather than forcing a model-specific or device-specific tile. - Gate the new candidates on explicit interleave4 packing capability, FP16/BF16 activation type, quantization/features, production alignment, and grid limits. No new physical architecture-ID eligibility gate is added. Existing hardware support and packing dispatch remain unchanged; native SM90/interleave1 weights do not use these geometries. - Profile eligible decode using rotating weights/scales beyond L2 when the matrix fits in cache. Use measured streaming time without the old heuristic CUTLASS cache penalty for this path. Keep the previous profiling behavior for other configurations. - Key profiled tactics by device, packing/kernel SM, dtype, quantization, group size, bias, and zero-point presence so incompatible configurations do not share a selection. - Use checked size/offset arithmetic and reserve the maximum scratch across actual initial M buckets. Charge lazy profiling only for its requested bucket; decode's rotating buffers are not added to lazy prefill allocations. - Add commented tests for geometry legality, grid boundaries, architecture-independent eligibility, cache separation, streaming scratch/overflow handling, FP16/BF16 CUTLASS-reference numerics, and workspace budgets. - The CPU AVX2 weight/scale prefetch addition remains removed. Existing upstream paired-K and wave-aware tactics are preserved; the M=1 hybrid coexists with them without changing their hardware eligibility policy. ### Motivation and Scope Improve INT4 decode without restricting performance tuning to an A100-derived shape envelope. Preserve INT4 defaults and INT4 lm_head, CUDA graphs, and existing fallback/packing contracts. No model export, tokenizer change, Foundry Local change, or llama.cpp change is included. ### A100 Performance Fresh measurements of the ported PR candidate on NVIDIA A100-SXM4-80GB (physical GPU6), symmetric INT4/group32/interleave4. **Before** is the preceding reviewed PR dispatch policy; **after** is the actual hybrid profiler-selected tactic. Both use identical current kernel math/layout; this is not a separately rebuilt old PR provider comparison. Median microseconds per operation, 24 samples per path. CUDA events time graph replay with at least 256 operations per sample and rotating execution order. Warm reuses one weight/scale matrix; streaming rotates at least 128MiB and exceeds twice device L2 (large matrices use two copies). Warm large matrices still exceed L2. One profiled tactic is used in both cache modes; no post-hoc fastest-tile selection is substituted. Each cell is **before -> after microseconds (latency change)**; negative percentages are faster. | M | N | K | FP16 Warm | FP16 Streaming | BF16 Warm | BF16 Streaming | |---:|---:|---:|---|---|---|---| | 1 | 1536 | 256 | 3.904 -> 2.868 (-26.5%) | 4.609 -> 2.689 (-41.7%) | 3.268 -> 2.356 (-27.9%) | 4.680 -> 2.694 (-42.4%) | | 1 | 256 | 1536 | 2.348 -> 2.348 (+0.0%) | 2.709 -> 2.708 (-0.0%) | 2.380 -> 2.380 (+0.0%) | 2.735 -> 2.735 (+0.0%) | | 1 | 512 | 1536 | 4.952 -> 2.508 (-49.4%) | 7.613 -> 2.937 (-61.4%) | 5.096 -> 2.540 (-50.2%) | 7.737 -> 2.963 (-61.7%) | | 1 | 2048 | 1536 | 5.140 -> 3.400 (-33.9%) | 7.992 -> 4.168 (-47.8%) | 5.320 -> 3.412 (-35.9%) | 8.120 -> 4.192 (-48.4%) | | 1 | 4096 | 1536 | 5.904 -> 4.292 (-27.3%) | 8.858 -> 5.504 (-37.9%) | 6.220 -> 4.312 (-30.7%) | 9.052 -> 5.596 (-38.2%) | | 1 | 6144 | 1536 | 6.104 -> 4.964 (-18.7%) | 9.464 -> 6.420 (-32.2%) | 6.372 -> 5.038 (-20.9%) | 9.648 -> 6.522 (-32.4%) | | 1 | 12288 | 1536 | 7.776 -> 7.580 (-2.5%) | 11.468 -> 10.694 (-6.7%) | 7.732 -> 7.304 (-5.5%) | 11.600 -> 10.504 (-9.4%) | | 1 | 262144 | 1536 | 141.078 -> 141.058 (-0.0%) | 140.428 -> 140.424 (-0.0%) | 141.444 -> 141.452 (+0.0%) | 140.848 -> 140.846 (-0.0%) | | 1 | 1536 | 2048 | 5.224 -> 3.400 (-34.9%) | 8.006 -> 4.100 (-48.8%) | 5.360 -> 3.412 (-36.3%) | 8.126 -> 4.096 (-49.6%) | | 1 | 1536 | 4096 | 4.288 -> 4.288 (+0.0%) | 5.800 -> 5.800 (+0.0%) | 4.340 -> 4.342 (+0.0%) | 5.862 -> 5.880 (+0.3%) | | 1 | 1536 | 6144 | 11.912 -> 5.392 (-54.7%) | 20.476 -> 7.456 (-63.6%) | 12.440 -> 5.416 (-56.5%) | 20.958 -> 7.512 (-64.2%) | | 1 | 1536 | 12288 | 21.940 -> 8.728 (-60.2%) | 38.664 -> 12.130 (-68.6%) | 22.996 -> 8.204 (-64.3%) | 39.662 -> 12.134 (-69.4%) | | 1 | 2880 | 4096 | 5.664 -> 5.832 (+3.0%) | 7.688 -> 7.372 (-4.1%) | 5.756 -> 5.756 (+0.0%) | 8.054 -> 8.078 (+0.3%) | | 1 | 5120 | 3072 | 6.344 -> 6.138 (-3.2%) | 8.962 -> 8.636 (-3.6%) | 6.508 -> 6.240 (-4.1%) | 8.988 -> 8.822 (-1.8%) | | 1 | 8192 | 4096 | 10.396 -> 10.284 (-1.1%) | 15.538 -> 15.666 (+0.8%) | 10.772 -> 10.468 (-2.8%) | 15.628 -> 15.622 (-0.0%) | | 1 | 8192 | 256 | 3.498 -> 3.012 (-13.9%) | 5.070 -> 3.512 (-30.7%) | 3.576 -> 3.036 (-15.1%) | 5.140 -> 3.544 (-31.1%) | | 1 | 524224 | 256 | 85.700 -> 64.634 (-24.6%) | 85.852 -> 64.600 (-24.8%) | 87.020 -> 63.096 (-27.5%) | 87.236 -> 63.142 (-27.6%) | | 1 | 524288 | 256 | 85.704 -> 64.490 (-24.8%) | 85.902 -> 64.582 (-24.8%) | 87.046 -> 62.914 (-27.7%) | 87.270 -> 62.904 (-27.9%) | | 2 | 2880 | 4096 | 7.520 -> 7.520 (+0.0%) | 10.784 -> 10.784 (+0.0%) | 8.084 -> 8.084 (+0.0%) | 10.848 -> 10.848 (+0.0%) | | 4 | 2880 | 4096 | 8.358 -> 8.356 (-0.0%) | 10.800 -> 10.804 (+0.0%) | 8.532 -> 8.532 (+0.0%) | 10.876 -> 10.880 (+0.0%) | Most targeted projections improve, but this is not a universal speedup: FP16 N=2880/K=4096 is approximately 3.0% slower warm and 4.1% faster streaming. Wide-N cases select CUTLASS and improve approximately 25-28%. M=2/4 controls are effectively unchanged. The N=262144/K=1536 projection retained the default tactic in this run. All 80 dtype/shape/cache cases passed analytical nonzero-output checks for all three measured paths (old geometry, reviewed dispatcher, selected hybrid); 5,760 timing records were collected. This analytical performance check is not comprehensive accuracy coverage; random CUTLASS-reference tests provide separate numerical coverage. Clocks were unlocked on a shared host; sub-percent differences are inconclusive. Timing was not run concurrently with local builds/tests. These are sampled kernel-level results, not end-to-end operator/model speedup claims or evidence for untested devices. ### A100 Validation - Ported PR-source incremental Release compact CUDA provider and internal test module built/linked; **48 focused A100 CUDA tests passed**, including geometry numerics, grid limits, packing routing, cache/eligibility, and workspace budget tests. - All six relevant C++ translation units passed compact-mode and full-mode syntax checks against the PR tree. Full-mode runtime is not claimed. - Modified FP16/BF16 INT4 dispatchers compiled for SM80 and SM90. This is not a complete SM90 provider build or an H100/H200 runtime test. - Eligibility tests use synthetic architecture IDs 75/80/86/89/90/100/120 with compatible/incompatible layouts. They prove the eligibility policy is layout-driven, not execution correctness/performance on those physical GPUs. - Pinned clang-format 20.1.8 and changed-file whitespace checks passed. - Incremental runtime validation reused the existing integrated host/core and unchanged CUDA objects, rebuilding the ported provider/test slice and dependent kernel registration. Tests used ABI-matching host/core headers with the PR CUDA-header overlay. This is not a fresh standalone full-tree PR build or a full-suite result. ### H200 Performance Fresh measurements at `33099b65b5d3ac69ae493a04c96ca6a1c97f6b11` on NVIDIA H200 (physical GPU1, SM90, 60MiB L2), symmetric INT4/group32/interleave4. This uses the SM80-compatible packing/kernel family on H200, not native SM90/interleave1 weights. Kernel architecture 80 identifies the packing contract, not a physical-device tuning gate. **Before** is the preceding reviewed PR dispatcher; **after** is the actual production-profiler-selected hybrid tactic. Both use identical current-head kernel math/layout, not separately rebuilt old/new providers. A tactic is selected once per dtype/shape and reused for both cache modes; no post-hoc fastest-tile selection is substituted. Median microseconds per operation, 24 samples/path. CUDA events time graph replay with at least 256 operations/sample and rotating path order. Warm reuses one weight/scale matrix; streaming rotates weights and scales across at least 128MiB and more than twice device L2, with at least two copies. Warm large matrices still exceed L2. Timing was not run concurrently with local builds/tests. Each cell is **before -> after microseconds (latency change)**; negative percentages are faster. | M | N | K | FP16 Warm | FP16 Streaming | BF16 Warm | BF16 Streaming | |---:|---:|---:|---:|---:|---:|---:| | 1 | 1536 | 256 | 2.498 -> 1.762 (-29.5%) | 3.914 -> 2.262 (-42.2%) | 2.538 -> 1.748 (-31.1%) | 3.972 -> 2.252 (-43.3%) | | 1 | 256 | 1536 | 1.815 -> 1.816 (+0.1%) | 2.317 -> 2.317 (-0.0%) | 1.824 -> 1.823 (-0.0%) | 2.326 -> 2.325 (-0.0%) | | 1 | 512 | 1536 | 3.844 -> 1.850 (-51.9%) | 6.338 -> 2.379 (-62.5%) | 3.954 -> 1.859 (-53.0%) | 6.451 -> 2.378 (-63.1%) | | 1 | 2048 | 1536 | 3.921 -> 2.258 (-42.4%) | 6.573 -> 2.789 (-57.6%) | 4.054 -> 2.274 (-43.9%) | 6.671 -> 2.803 (-58.0%) | | 1 | 4096 | 1536 | 4.055 -> 2.796 (-31.1%) | 6.763 -> 3.466 (-48.7%) | 4.179 -> 2.754 (-34.1%) | 6.854 -> 3.473 (-49.3%) | | 1 | 6144 | 1536 | 4.547 -> 3.248 (-28.6%) | 7.046 -> 3.887 (-44.8%) | 4.847 -> 3.252 (-32.9%) | 7.198 -> 4.089 (-43.2%) | | 1 | 12288 | 1536 | 5.024 -> 4.652 (-7.4%) | 7.716 -> 5.644 (-26.8%) | 5.311 -> 4.782 (-10.0%) | 7.911 -> 5.949 (-24.8%) | | 1 | 262144 | 1536 | 69.827 -> 64.180 (-8.1%) | 69.840 -> 64.167 (-8.1%) | 71.753 -> 62.962 (-12.3%) | 71.760 -> 63.026 (-12.2%) | | 1 | 1536 | 2048 | 3.964 -> 2.247 (-43.3%) | 6.692 -> 2.809 (-58.0%) | 4.080 -> 2.273 (-44.3%) | 6.796 -> 2.823 (-58.5%) | | 1 | 1536 | 4096 | 2.849 -> 2.848 (-0.0%) | 3.812 -> 3.813 (+0.0%) | 2.908 -> 2.908 (+0.0%) | 3.853 -> 3.858 (+0.1%) | | 1 | 1536 | 6144 | 9.227 -> 3.439 (-62.7%) | 16.332 -> 4.667 (-71.4%) | 9.622 -> 3.524 (-63.4%) | 16.694 -> 4.752 (-71.5%) | | 1 | 1536 | 12288 | 17.001 -> 5.235 (-69.2%) | 30.900 -> 7.279 (-76.4%) | 17.786 -> 5.412 (-69.6%) | 31.686 -> 7.446 (-76.5%) | | 1 | 2880 | 4096 | 3.494 -> 3.495 (+0.0%) | 4.451 -> 4.456 (+0.1%) | 3.566 -> 3.567 (+0.0%) | 4.501 -> 4.498 (-0.1%) | | 1 | 5120 | 3072 | 4.106 -> 3.881 (-5.5%) | 5.223 -> 5.118 (-2.0%) | 4.111 -> 3.937 (-4.3%) | 5.235 -> 5.206 (-0.6%) | | 1 | 8192 | 4096 | 6.720 -> 6.120 (-8.9%) | 8.353 -> 7.482 (-10.4%) | 6.928 -> 6.487 (-6.4%) | 8.436 -> 7.886 (-6.5%) | | 1 | 8192 | 256 | 2.613 -> 2.102 (-19.5%) | 4.008 -> 2.531 (-36.8%) | 2.724 -> 2.023 (-25.7%) | 4.097 -> 2.554 (-37.7%) | | 1 | 524224 | 256 | 53.505 -> 39.812 (-25.6%) | 53.509 -> 39.821 (-25.6%) | 54.786 -> 39.209 (-28.4%) | 54.795 -> 39.209 (-28.4%) | | 1 | 524288 | 256 | 53.487 -> 39.796 (-25.6%) | 53.493 -> 39.794 (-25.6%) | 54.779 -> 39.188 (-28.5%) | 54.784 -> 39.195 (-28.5%) | | 2 | 2880 | 4096 | 6.042 -> 6.040 (-0.0%) | 8.734 -> 8.728 (-0.1%) | 6.087 -> 6.088 (+0.0%) | 8.563 -> 8.559 (-0.0%) | | 4 | 2880 | 4096 | 5.750 -> 5.749 (-0.0%) | 7.264 -> 7.269 (+0.1%) | 5.854 -> 5.857 (+0.0%) | 7.351 -> 7.357 (+0.1%) | Measured latency changes span -76.5% to +0.1%, with no material regression in this run. Wide-N K=256 cases select CUTLASS and improve approximately 26-29%; N=262144/K=1536 improves approximately 8-12%. M=2/4 controls are effectively unchanged. Clocks/power settings were unchanged on the shared host; maximum sample coefficient of variation was 1.12%, and sub-percent differences are inconclusive. These are sampled kernel-level results, not end-to-end operator/model speedup claims. All **80 dtype/shape/cache cases** passed exact nonzero analytical checks for all three measured paths (old geometry, reviewed dispatcher, selected hybrid); **5,760 raw timing records** were independently verified against the reported medians. The final fixture uses the first 256 activations as one, remaining activations as zero, scales as one, and packed weight 0x99, giving expected output min(K,256) while retaining full matrix traversal. The initial all-one activation fixture at K=12288 returned 8192 instead of 12288 in the legacy BF16 path; it was not used for the final table. This bounded performance fixture is not comprehensive accuracy coverage; random CUTLASS-reference tests provide separate numerical coverage. Because the H200 fixture is bounded, these tables are not a controlled cross-device A100-versus-H200 comparison. #### H200 Validation - Current-head native Release compact CUDA provider and internal test module rebuilt successfully. **17 focused native tests passed**: nine policy/geometry/cache tests and eight FP16/BF16 INT4 numerical tests, including profiled geometries against random CUTLASS references, grid limits, and Hopper packing routing. No full-suite or full-mode runtime claim is made. - The standalone benchmark compiles unchanged production profiler/kernel sources with a local bridge-only compatibility header supplying native ORT logging/ONNX types, and links current-head native host support libraries. No tracked production edits were made for this measurement. - The final harness and production FP16/BF16 INT4 dispatchers compile for SM80 and SM90. Synthetic eligibility tests cover architecture IDs 75/80/86/89/90/100/120 with compatible/incompatible layouts. Geometry eligibility is packing/shape/alignment-driven, not physical-architecture-ID-driven; existing hardware support, packing routes, and fallbacks remain unchanged. These checks do not claim physical runtime/performance on untested GPUs. The [previous H200 results](#33177 (comment)) apply to the older conservative-dispatch revision `892ef02a16bab34e5e31aea12947dea859d609c8` and remain historical evidence, not validation of this hybrid update. H100 runtime/performance is not claimed. Windows, ROCm, other physical GPU architectures, native asymmetric SM90-layout runtime, and full-suite coverage are not claimed. Production alignment restrictions remain; unsupported widths such as N=524296 are not newly enabled. Reviewer confirmation remains pending. Benchmark harnesses, raw measurements, generated models, binaries, dependencies, and caches are excluded from this PR. No new CPU optimization claim is made. Current update: `33099b65b5d3ac69ae493a04c96ca6a1c97f6b11`. Merged upstream `a477effab7` and resolved shared profiler/dispatcher/cache conflicts. The A100 table was measured on `c1665cc3de` before this merge; timings were not rerun after conflict resolution. ### Conflict Resolution Validation Upstream paired-K variant 1, wave-aware dispatch flags, and their cache-key fields are retained alongside decode variants 2-7 and device/quantization keys. A combined regression checks M=1 candidates and CUTLASS fallback across paired/wave modes, with paired-K restricted to M=5-8. All 48 focused A100 tests passed; six full-mode C++ syntax checks, SM80/SM90 dispatcher compilation, and pinned formatting/whitespace checks passed. Runtime validation remains incremental compact-mode using an ABI-matched existing host/core, not a fresh full-tree build. Current-head H200 runtime results and validation scope are reported above.
## Summary Follow up on #32794 with an optimized **routed expert** matmul for block-scaled FP8 QMoE. This is separate from the dense MatMul kernel in #33074. - Stage selected-expert E4M3 weights and FP16 activations into shared tiles and use supported 16x16x16 FP16-input subgroup-matrix instructions with FP32 accumulation. FP8 is decoded before the matrix instruction; this is **not native FP8 matrix hardware**. - Handle per-128x128 scales, signed FP8 codes, FC1/FC2 bias, odd-byte expert boundaries, and incomplete matrix/scale tiles. - Only dispatch the matrix path for direct expert batches with at least 128 rows and 256 input/output channels on adapters exposing the required subgroup configuration. Preserve the original scalar path for single-token decode, smaller routed groups, and other adapters. A 16-row synthetic case regressed with the matrix path, so it is deliberately excluded. ## Validation - Linux Release static WebGPU provider, provider tests, and Python WebGPU runtime built successfully. - All 17 focused WebGPU QMoE tests passed on NVIDIA H200/Vulkan, including a new 128-row, 257-channel case covering signed FP8 values, distinct 128x128 block scales, bias, tail tiles, and non-word-aligned expert offset. - Synthetic end-to-end QMoE with 128 rows, 2560 hidden channels, 640 intermediate channels, two experts, and 30 repeated calls: matrix path 4.4–5.1 ms/call versus original scalar 6.17 ms/call in this harness (about 1.2–1.4x). This includes routing and other kernel work; it is **not isolated matrix-kernel timing**. At 16 rows the original scalar path remains enabled (about 3.7 ms/call). - `clang-format --dry-run --Werror` and `git diff --check` passed. No official block-FP8 checkpoint or end-to-end model-quality claims: the local checkpoint is BF16. Stack base is #32794 for review; merge the PRs individually when ready. ## Stack update (October 5) Restacked onto #32794 after its separate-FC3 regression was added. The review diff remains limited to the three QMoE optimization files. All 18 focused stacked WebGPU QMoE tests passed on H200/Vulkan; CI is rerunning for the updated base. ## Current main update (October 5) Merged the latest #32794 base, which itself now includes current `main` and merged INT2/mixed-width QMoE support. The review diff remains the three tiled-FP8 files. All 15 focused H200 block-FP8 and merged INT2 compatibility tests pass; new-head CI is running. ## Review follow-up (October 6) - Apply FP32 block scales to FP32 subgroup-matrix partial sums, not FP16 staged weights; a +448/-448 cancellation test with scale 200 checks finite output. - Keep matrix workgroup dimensions within the adapter limit, falling back to scalar when its dispatch is legal; reject oversized scalar dispatch through the updated #32794 base. - Add mixed 128+4 row routing with distinct expert FC1/FC2/FC3 weights, scales, and nonzero biases, an independent SwiGLU reference, and an adapter-aware assertion that all three eligible projections actually dispatch the matrix path. - After merging #32794 head `40cebdeeef`, built the Linux Release provider tests and passed all 30 focused WebGPU QMoE tests on H200/Vulkan; formatting and `git diff --check` passed. This does not imply other platform CI has completed. ## October 6 base update Merged updated FP8 QMoE base 09737a3, inheriting orphan FC3-scale validation and synchronized operator docs that address the Windows GPU documentation failure (stacked head 23922ce). All 31 focused H200 WebGPU QMoE tests pass; fresh CI is pending. Previously drafted replies to three review threads have now been published. ## October 7 base update Merged updated #32794 base eb51a28 to inherit explicit rejection of unused FP8 activation-scale inputs. The new four-case regression and all 32 focused H200 WebGPU QMoE tests pass on stacked head 158c7d0. Fresh CI is pending. ## October 7 CI stack update Merged current main and updated #32794 (541e5c7), including the already-merged upstream CUDA GQA `is_int8` build fix #33179. The static WebGPU provider tests build and all 32 focused H200 routed-QMoE tests pass. New-head CI is pending. ## October 7 upstream CI fix merged Restacked on updated #32794 at 79873ae, including merged upstream #33168 ONNX node-test type conversion. Static WebGPU provider tests build and all 32 focused H200 routed-QMoE cases pass. Fresh CI is pending; previous-head failures are not evidence for this head. ## October 8 merged-base refresh and paired performance Merged current main after #32794 landed. The review diff remains the QMoE matrix-path implementation and its tests. In an integrated branch with #33058 and #33074, 52/52 H200 WebGPU QMoE and dense FP8 regressions passed. For a synthetic FP16-input, 2-expert block-FP8 QMoE (128 tokens, hidden 2560, intermediate 640), ONNX Runtime perf_test on H200/Vulkan measured 7.86 ms/run on the merged #32794 implementation versus 6.05 ms/run with the tiled path (median of three 100-run trial averages, ~1.30x). These are whole-session inference latencies, not isolated kernel timings or an official-FP8-checkpoint result. Current-head CI is pending. ## October 9 main refresh and subgroup ownership fix Merged main at 9a6089e, including merged #33058 and its gate tests/documentation. The PR now targets the merged FP8 QMoE implementation on main; #32794 is no longer an unmerged stack prerequisite. Addressed the new subgroup-ownership review in 7567ecd: the shared shader entry point exposes the actual `@builtin(subgroup_id)` as `subgroup_index`, and the QMoE matrix path uses it for accumulator initialization, left-operand loading, and result storage. Cooperative tile transfers still use local invocation indices. This matches the dense FP8 implementation in #33074. Validation on the updated head: - Linux Release WebGPU provider-test build succeeded. - 38/38 focused H200/Vulkan WebGPU QMoE tests passed, including the matrix-capability-aware mixed-routing/separate-FC3 regression and the merged gate cases. - 89/89 WGSL-template tests passed. - Captured GPU shaders use the subgroup-ID builtin for all three matrix tile offsets; no `lane / 32u` offset remains. - `clang-format --dry-run --Werror` and `git diff --check` passed. The subgroup-mapping fix was executed on H200, not an adapter with a noncontiguous subgroup mapping. Fresh GitHub CI is pending; the local results are not a claim that all platform CI has completed. --------- Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
## Summary Complete the preparation-sizing half of Stage 1 in the policy-based, route-aware workspace-estimation design (#32944). This combines the agreed QKV-preprocessing refactor and the remaining cache/sequence preparation sizing into one PR. Builds on #32945, now merged into `main`, which consolidates MEA, unfused, XQA and Flash backend scratch sizing. - Share checked cache-row, past-preservation, staging, compaction, sequence-vector and QKV-preprocessing calculations between `ComputeInternal` and `GetGQAPreparationRecipe`. - Remove the duplicate `GQABufferRequirements` implementation. - Keep the existing separate allocations, ownership, pointer wiring, dispatch priorities and lifetimes. - Preserve the preparation recipe's aligned layout and full validation; the live kernel calls narrow sizing helpers rather than inheriting unrelated recipe-admission restrictions. This is sizing-code consolidation, not a missing-buffer fix. It does not activate dispatch policies, exclude routes from L1, coalesce allocations, or change static-preallocation strategy. ## Compatibility details - Cache sizing runs before backend selection and retains the original present-cache capacity. Windowed staging still changes the runtime extent from `C` to `C + S`. - Past preservation uses the aliased tensor's physical row width; staging uses the physical output row width; compaction retains its packed-row interpretation. INT4 rows are not packed a second time. - QKV sizing runs after the existing backend selection and receives the effective post-staging capacity. cuDNN retains the existing generic Q-materialization behavior. - Flash fast decode still reuses the device sequence-length input instead of allocating sequence vectors, and still suppresses the QKV-preprocessing buffer. - Existing healthy-input byte counts are retained. Arithmetic overflow and invalid sizing inputs produce explicit sizing statuses at the runtime boundary. ## Coverage and validation - Fresh standalone build of the existing preparation Google Test source: **50 tests passed**, including the legacy QKV feature matrix, dense/byte/INT4 cache geometry, physical packed rows, sequence-vector suppression, invalid dimensions and overflow. - MSVC `/W4 /WX` compilation of the actual CUDA GQA runtime translation unit in both shared-provider and plugin-adapter configurations, with Flash, MEA, FP8 and INT4 enabled. - MSVC `/W4 /WX` compilation of the graph-time estimator and XQA/Flash sizing-test translation units. - MSVC `/W4 /WX` compilation of the CUDA GQA operator-test translation unit. - Four new CUDA regressions compare packed and separate QKV on unfused/Flash paths, including windowed staging. They reuse the aliasing helper, which disables CPU fallback and verifies actual backend dispatch. - `clang-format --dry-run --Werror` and `git diff --check` passed. **Full CUDA inference execution is not validated locally against a freshly linked provider.** The new operator regressions and existing GQA tests need full CUDA build/test CI. ## Stack Base: `main`. The conflict-resolution merge retains the early preparation-sizing initialization, the upstream seq-free eligibility helpers, and the INT8/FP8 storage-width normalization. The envelope and policy-reader PRs (#33014, #33016 and #33042) remain independent of this refactor. Committed-policy resolution, coverage/envelope enforcement, bounded fallback and policy-bound estimates remain follow-up work. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7fa98d07-aec3-41b7-92b8-b0ec317a95bf
…and serialized execution (#33082) ### Description <!-- Describe your changes. --> Restore WebGPU plugin compatibility starting with **ORT 1.24.4**, while retaining **Session-owned command recording in both execution modes**. The additional real-host matrix below found an unresolved **ORT 1.26.0 host-side rejection**; this is not a claim that every intervening release currently works. - Each Session exclusively owns its recording through `unique_ptr`; there is no context-owned shared legacy recording. - Select the older-host compatibility policy with `UseSerializedExecutionMode()` and `ShouldUseSerializedExecutionMode()`, now isolated in `ep/runtime_compatibility.h/.cc` rather than the stream implementation. - In serialized mode, Session `Alloc` and matching `AllocOnStream` use `!IsRunActive()` to submit cached-buffer clears outside Run and defer them during Run. Kernel scratch uses ordinary Tensor allocation through the same allocator, without an `AllocForKernel` API or the unavailable old-host `KernelContext_GetSyncStream`. - Env allocations and concurrent-mode ordinary Session allocations retain independent, immediately submitted clears. Cached-buffer clearing and kernel dispatch batching remain enabled. - For streamless framework copies, submit pending active-Session work before recording the copy independently. The context borrows a non-owning pointer to that recording; it does not own or share the command state. Session teardown discards only that Session's pending entries. - Remove the context-wide Run gate and rely on the required caller-serialization contract. `SerialRunsAfterRejectedStreamOverride` covers a rejected stream override followed by successful serial Runs on both the affected Session and another Session. | Host ORT version | Default execution mode | | --- | --- | | 1.26.0 | Currently blocked by the host's plugin-version check; see validation below | | 1.28.x starting at 1.28.3 | Concurrent independent Sessions | | 1.30.x starting at 1.30.1 | Concurrent independent Sessions | | 1.31 and later | Concurrent independent Sessions | | Other supported versions, including all 1.29.x | Serialized compatibility | Set `ORT_WEBGPU_EP_FORCE_LEGACY=1` **before plugin registration** to force serialized compatibility. The existing override and `onnxruntime_webgpu_legacy_test` CTest name are retained for compatibility; neither implies shared recording. The override cannot enable concurrency on an unsupported host. **Serialized mode requires callers to serialize all WebGPU operations on the same device**, including Session creation/destruction, Run, allocation, I/O binding, and Env transfers. Use sequential graph execution. This is not a fixed-CPU-thread requirement, and the plugin neither detects nor serializes overlapping operations. Callback reentry remains unsupported. #### Input/output copy ordering ORT 1.24.4 forwards supplied streams through batched `CopyTensors`; its separate single-tensor wrapper drops the stream. `BindInput` uses a single-input copy, and mixed source/destination device pairs can make a batch fall back to individual copies. | Path | Ordering mechanism | | --- | --- | | `BindInput` outside Run, CPU to GPU | Allocation submits any cached-buffer clear before returning, including through `AllocOnStream`. A subsequent streamless upload is ordered after it on the device queue; no active Run pointer is needed. | | Input copy during Run, CPU to GPU | With a stream, the clear and upload use the same Session recording. Without a stream, flush the active Session first, then upload using a local recording. | | Output copy during Run, GPU to CPU | Submit the producer before readback, either through its explicit stream or the active-Session pre-copy flush. This does not depend on `OnRunEnd`, which occurs after output copies. | | Mixed-direction inputs/outputs | Each streamless copy follows the same submission boundary if the old host falls back to individual copies. | | Streamless GPU-to-GPU copy within one context | Submit preceding active Session work, then submit the local copy before returning. Cross-context copies remain unsupported. | These are submission boundaries, not per-kernel CPU waits. Readback retains its existing completion wait. Compared with a shared recording, a streamless copy with pending Session work can require an additional queue submission; performance equivalence has not been established. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Allow independently distributed WebGPU plugins to support older applications without requiring the host-side single-copy stream-forwarding fix in #32666. Builds on #32874. Following review, the implementation now keeps Session-owned recording rather than introducing a second ownership model for old hosts. A real ORT 1.24.4 mixed-output repro demonstrated the missing submission boundary: an independent GPU Neg branch fetched to CPU returned 0 instead of -1 when the Session's pending computation was not submitted before a streamless readback. The other independent CPU branch was fetched to GPU, triggering mixed-direction single-copy fallback. Both targets and the GPU input were preallocated. Adding the boundary makes both output orderings pass; this shows the need for ordering coordination, not shared recording ownership. The alternative developed in #33228 has been incorporated here, and that duplicate PR is closed. This revision includes upstream `main` at `e19a9ae831`. **Scope:** Removing the context-wide gate avoids permanently blocking other Sessions when a host error skips `OnRunEnd`. It does not guarantee recovery of all unfinished Run/capture state. Clarifying the host's `OnRunStart`/`OnRunEnd` error-path contract and complete failed-replay recovery remain separate follow-up work. ### Validation Validated commit **`20ff4e2f63`** on Windows x64 Release with the shared WebGPU plugin and EP API adapters. Current-host tests use ORT **1.32.0**. `W:` was a temporary mapping to this worktree and was removed afterward. **Build: passed.** ```powershell cmake --build W:\build\WebGpuCompat\Release --config Release --target onnxruntime_providers_webgpu onnxruntime_provider_test --parallel 8 -- /nologo /verbosity:minimal ``` **Serialized compatibility: 41 executed tests passed, no failures or skips.** CTest sets `ORT_WEBGPU_EP_FORCE_LEGACY=1` before loading the plugin and selects serial-safe cases. ```powershell Set-Location W:\build\WebGpuCompat\Release ctest -C Release -R '^onnxruntime_webgpu_legacy_test$' --output-on-failure ``` **Concurrent-mode selection: 57 executed tests passed, no failures or skips.** Two pre-existing disabled diagnostic tests were not enabled in either selection. ```powershell Set-Location W:\build\WebGpuCompat\Release\Release $env:ORT_UNIT_TEST_MAIN_DYNAMIC_PLUGIN_EP_CONFIG_JSON='{"ep_library_registration_name":"WebGPU_PluginEP","ep_library_path":"onnxruntime_providers_webgpu.dll","selected_ep_name":"WebGpuExecutionProvider"}' $env:ORT_WEBGPU_EP_FORCE_LEGACY='0' .\onnxruntime_provider_test.exe '--gtest_filter=WebGpuPluginRecordingModeTest.*:WebGpuPluginSharedAllocatorRegistrationTest.*:WebGpuPluginSharedAllocatorTest.*:WebGpuSessionAllocatorDeathTest.*:PluginEpWebGpuConcurrency.*' ``` Coverage includes version boundaries, allocation-clear submission, scratch batching, serial and concurrent capture/replay, mixed feeds/fetches, and serial Runs after a rejected stream override. **Actual ORT 1.24.4: all 9 local-harness scenarios passed, 20 iterations each.** The standalone API-24 C++ probe loaded the official 1.24.4 DLL and the plugin rebuilt from this commit; profiles confirmed CPU/WebGPU node placement. Cases cover CPU I/O, dirty-buffer BindInput, mixed-direction inputs and outputs in both orders, GPU-to-CPU-to-GPU graph execution, scratch reuse, and capture-enabled execution. The following invocation used the existing local `recording-ownership-probe` directory (`$probe`); this harness is not part of the repository or CI: ```powershell $variant='serialized-policy-20ff4e2f63' $dest=Join-Path $probe $variant $env:ORT_WEBGPU_EP_FORCE_LEGACY='1' python "$probe\run_probe.py" $variant "$dest\onnxruntime_providers_webgpu.dll" ``` **Capture limitation:** the actual 1.24.4 run exercised initial capture and its internal replay. Its subsequent Runs did not exercise host-driven replay, because that host lacks the plugin capture/replay callbacks. Repeated host-driven replay was covered by the current-host tests above. Other operating systems and performance benchmarks were not tested in the above `20ff4e2f63` validation. #### Additional local validation at `62e3504c2e` Removed only the redundant pre-capture `OnRunStart` flush. `CaptureBegin` still flushes before entering capture; streamless-copy, Run-end, and standalone replay submission boundaries are unchanged. The plugin rebuilt successfully, and the three focused serialized capture tests passed: ```powershell $env:ORT_UNIT_TEST_MAIN_DYNAMIC_PLUGIN_EP_CONFIG_JSON='{"ep_library_registration_name":"WebGPU_PluginEP","ep_library_path":"onnxruntime_providers_webgpu.dll","selected_ep_name":"WebGpuExecutionProvider"}' $env:ORT_WEBGPU_EP_FORCE_LEGACY='1' .\onnxruntime_provider_test.exe '--gtest_filter=PluginEpWebGpuConcurrency.SerialGraphCaptureWithCpuFeedsAndFetches:PluginEpWebGpuConcurrency.SerialSingleSessionMultipleGraphCaptureIds:PluginEpWebGpuConcurrency.GraphCaptureReplayInterleavedWithIdleSessionCpuBindInput' ``` A separate local matrix used **one unchanged plugin DLL** with official Windows x64 cores 1.24.4, 1.25.1, 1.26.0, 1.27.1, 1.28.3, 1.29.1, and 1.30.0, plus the local 1.32.0 Release core. Loaded module paths, actual core versions, and plugin hashes were checked. - **Native copy/allocation/capture probe: 63 scenarios passed, 9 failed**, with 20 iterations per scenario. All nine scenarios passed on every tested core except 1.26.0. That core rejects `OrtEp::ort_version_supported=32` because its kernel-creation check requires a value no greater than 26. This host restriction remains unresolved; forcing serialized mode does not address it. - **Capture support differs from execution-mode eligibility.** In the simple native capture probe, 1.24.4 and 1.25.1 performed one initial internal replay; the other passing cores performed repeated host-driven replay. - **Local Phi-4 workload, capture OFF:** all seven usable cores passed fresh-generator repeatability, serially interleaved-generator checks, multi-turn checks, and a repeated exact 1024-input/300-output-token workload. An isolated GenAI build compiled against API 24 was held fixed across cores; the normal installed package requires API 29. Performance measurements used two warmups and five measured runs, generator reuse and shared KV buffers, with profiling disabled. - **Capture ON is not valid for this model graph.** It contains an `If` node. Tested cores from 1.26 onward reject capture at Session initialization. On 1.24.4 and 1.25.1, initialization succeeds but generated tokens differ after rewinding and repeating the same prompt. This also reproduces without profiling and with the pre-cleanup `20ff4e2f63` plugin, so it was not introduced by removing the redundant flush. No capture-ON throughput is reported for this model. - **Performance interpretation:** default core graph optimizations were retained; newer cores use additional MLP fusion. Cross-version throughput is therefore not an isolated measurement of serialized-mode overhead. Generated tokens were repeatable within each core but not identical between the older and newer optimizer groups. The matrix scripts and model are session-local, not checked into the repository or run by CI. `$matrix` below refers to the isolated `phi4-core-matrix` artifact directory: ```powershell python "$matrix\run_native.py" # failed overall: 63 passed, 9 failed on 1.26.0 python "$matrix\run_phi4.py" --phase smoke # failed overall: 7 passed, 9 unsupported/failing combinations python "$matrix\run_phi4.py" --phase correctness # passed: 21 checks; failing smoke combinations excluded python "$matrix\run_phi4.py" --phase benchmark # completed: 7 capture-OFF configurations; failing smoke combinations excluded ``` --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b6fbaacb-9846-4330-8fb3-2f0b13737d48 Copilot-Session: 8ca4e5fc-8c08-46bd-82a9-5cff5108efed
Bump plugin ep versions: Cuda 0.4.0 WebGPU 0.6.0 Also mark compile API boundary of version 31.
### Description - Introduce `MatMulAlgorithm` with six explicit implementations: subgroup matrix, GEMV, naive, subgroup, packed, and packed Split-K. - Add `MatMulExecutionPlanner`, which selects an algorithm and typed tuning configuration to produce an immutable `MatMulExecutionPlan` for each invocation. - Allow vendor planners to override performance policy and tuning while retaining common fallback behavior. - Route MatMul, pointwise Conv, and contrib Attention through `MatMulComputeDispatcher`, which validates plan/configuration compatibility and runtime prerequisites before dispatch. - Keep forced-algorithm selection internal to tests through `WebGpuExecutionProviderTestOptions` and the `CreateForTesting` factory path; no public session or provider option is added. - Preserve shared adapter-specific Split-K profiles in `WebGpuContext` so GEMM and MatMul consume the same immutable policy. - Integrate planner-controlled packed tuning with FP32 accumulation, Split-K FP32 reduction/cast-back, and dispatch-normalization safety. - Document the architecture in `docs/design/webgpu_matmul_execution_planner.md`. ### Motivation and Context WebGPU MatMul chooses among several implementations using shape, capability, and vendor-specific conditions. Keeping these choices as distributed conditionals made individual implementations difficult to test and vendor thresholds difficult to evolve. This refactor separates immutable problem facts, pure planning policy, typed execution configuration, runtime validation, and dispatch. A vendor can override performance boundaries or tune execution parameters without changing `MatMulComputeDispatcher`; returning no selection or configuration delegates to common defaults. ### Behavior and Compatibility - The common planner preserves subgroup-matrix precedence, adds the vendor-neutral GEMV specialization, retains the small-matrix Naive rule, uses existing Split-K policy, and otherwise falls back to packed MatMul. - GEMV automatic selection uses its measured N/K tuning window; forced tests may bypass those thresholds while retaining hard shader constraints. - Intel preserves its existing subgroup and Split-K architecture policies and measured boundaries. - Subgroup execution accepts only shader-supported sizes 8, 16, and 32, including vendor-provided configurations. - Automatic selection always uses Naive when K is zero before consulting vendor policy. Forced unsupported algorithms fail explicitly rather than silently falling back. - Packed vendor tuning is checked against scalar/vec4 shader contracts and overflow-safe dispatch arithmetic before device work. - Existing subgroup-matrix tiling remains in its dedicated selector and implementation object. - NVIDIA Split-K policy from #32333 is not included; this PR provides the extension point it can use. ### Validation - Built static and plugin WebGPU provider targets on Windows. - Ran planner, configuration, prerequisite, forced-algorithm, GEMV, Split-K, dispatch-normalization, recording, Attention, and pointwise Conv tests. - Ran upstream FP32 accumulation regression tests across MatMul, Gemm, Split-K, Intel subgroup, scalar/vec4 packed paths, and Conv. - Added numerical coverage for supported non-default packed tuning, rejection coverage for unsupported tuning/subgroup sizes, dynamic-shape reselection, and batched partial Split-K dispatch. - Hardware-specific tests inspect adapter capabilities and skip unsupported algorithms rather than forcing a backend.
…32605) ### Description [WebGPU] Widen MatMulNBits SubgroupMatrix-8x16x16 tiling to 128x256 - Parameterize kTileM and kTileN in the GPU kernel to support dynamic tiling configurations based on workload size. - Increase the kernel tile size from 64x64 (256 threads), to 128x256 (512 threads) for larg M to improve performance. - Centralize the selected tiling policy into a SubgroupMatrixMatMulNBitsTiling struct. **Test on Phi-4-mini, Intel Panther Lake:** | | Base Prefill TPS | Opt Prefill TPS | Improve | |---|---|---|---| | prompt-128 | 1902.96 | 1838.73 | -3% | | prompt-256 | 2182.59 | 2391.95 | 10% | | prompt-512 | 2133.16 | 2370.76 | 11% | | prompt-1024 | 2111.78 | 2312.07 | 9% | | prompt-2048 | 1864.42 | 2173.03 | 17% | | prompt-4096 | 1447.15 | 1748.95 | 21% | ### Motivation and Context See above.
### Description Prepack constant ConvTranspose1D and ConvTranspose2D weights during session initialization so repeated inference reuses the layout expected by the existing 2D shader instead of transposing the same weights on every run. - Save an owned copy of the original ONNX filter shape for validation and output-shape inference, removing inverse-permutation reconstruction for all packed layouts. Both 1D and 2D weights use HWIO; the saved shape preserves their original ranks. The 1D representation retains the synthetic height of one used by the 2D kernel. - Preserve per-run transposition for dynamic weights and when prepacking is explicitly disabled, and preserve the existing 3D prepacking path. - Correct two existing NHWC weight-vector accesses for grouped convolution with one output channel per group: their input-channel offsets must be expressed in vector units. The expanded tests reproduced incorrect results on the unmodified base, including dynamic weights. - Extend the existing WebGPU fixture with 176 parameterized tests covering both spatial ranks, layouts, FP32/FP16, groups/depthwise, scalar/vec2/vec4 and mixed widths, channel remainders, bias, stride/dilation/padding, initializer/dynamic weights, repeated runs, and prepack-count assertions. Retain all 176 existing ConvTranspose3D cases. Additional coverage checks genuine height-one 2D kernels and all 12 initializer/dynamic input-weight rank-mismatch combinations. ### Motivation and Context Follow-up to the non-blocking prepacking comment on #32688: #32688 (comment) ### Validation - Fresh Windows Release/D3D12 build of commit `96f564b873` with 12 parallel workers. - Final `onnxruntime_provider_test --gtest_filter="*ConvTranspose*"`: **430 passed, 4 expected CUDA/DML-only skips**, including all 176 new parameterized tests and all 176 existing 3D cases. CPU fallback is disabled for the WebGPU fixture. - All 17 focused shape-regression tests passed; the rank-mismatch test verifies rejection from the running WebGPU kernel and asserts prepack counts. - Before the implementation, 72 parameterized tests failed their missing-prepack-count assertion. The expanded grouped single-output-channel cases also exposed a numerical error in the original shader; the final run passes both initializer and dynamic variants. - C++ lintrunner and `git diff --check` passed. - Synthetic benchmark of the prepacking implementation at `edc3a6a0cc` on an NVIDIA GeForce RTX 5080: median warmed end-to-end Run speedups of **1.00x-1.18x** across 16 1D/2D, layout, dtype, and shape scenarios. One small FP16/NHWC case was effectively unchanged; the other 15 measured 1.06x-1.18x. Warm-context session initialization increased by about **0.06-0.14 ms**. - Benchmark uses the same initializer model with `session.disable_prepacking` toggled, verifies equal outputs and prepack counts, alternates ordering, and aggregates 25 samples per mode/scenario from five processes. Each sample averages 60 timed Run calls after 15 warmups, including host input/output transfers. These are single-operator measurements on one GPU, not a general model-speedup claim. The benchmark harness is a local artifact, not part of the source diff. - Browser/WASM, plugin builds, GPU capture mode, and full-model performance were not validated locally.
### Description Fix five native WebGPU shader-cache collisions using the existing metadata and cache hints: - **BiasAdd / BiasSplitGelu:** key one input's program type, which captures both element type and vector width. - **LayerNormalization:** distinguish mean-only from inverse-standard-deviation-only outputs. Output count already distinguishes the other combinations. - **InstanceNormalization:** include the epsilon literal embedded in the shader. - **Boolean And:** distinguish full-vector reads from single-lane broadcasts, which both use Boolx4 storage. Also correct BiasSplitGelu's FP16 division operand type so the correctly selected shader compiles. Add six small GPU regression tests. Each puts conflicting variants in one session, covers both node insertion orders, and repeats inference for warm-cache reuse. CPU fallback is disabled and full WebGPU validation is enabled. ### Motivation and Context This extracts the confirmed correctness fixes from #33153 so they can land before the proposed cache-framework migration. The DequantizeLinear fix is already merged in #33122. ### Validation - Fresh Windows Release build with native WebGPU/D3D12, based on `6e0f154e32`. - All six new tests fail on unmodified main and pass with the fixes; about 2.5 seconds locally. - With the FP16 arithmetic correction but without BiasSplitGelu's type key, five tests pass and BiasSplitGelu still fails numerically, independently confirming that collision. - 201 affected tests pass, zero skipped, with two BERT tests rerun after staging their missing model fixtures. Broad counts include CPU execution. - All 14 original numerical reproduction checks pass, including dynamic Boolean shape swaps and cross-session cache reuse, with CPU fallback disabled. - Clang-format, lintrunner, and `git diff --check` pass. Metal, WASM and plugin builds were not run locally.
### Description Remove the Intel Math Kernel Library license section from `ThirdPartyNotices.txt`. Intel MKL is no longer an ONNX Runtime dependency: MKLML support and its build option were removed by #6212, and the current dependency manifest does not include MKL. Tests were not run because this is a documentation-only change. ### Motivation and Context The stale notice incorrectly suggests that current ONNX Runtime distributions incorporate Intel MKL. Fixes #33215. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
### Description
Fix Python WebGPU provider creation in plugin hosts. Building with
`--use_webgpu shared_lib --build_wheel` previously referenced the
unavailable built-in WebGPU factory, causing Windows link errors.
Provider-name selection also stopped at the dedicated WebGPU branch
before reaching registered plugins.
The WebGPU branch now uses the built-in factory only when that
implementation is compiled in. Otherwise, non-minimal hosts use the
existing registered-device factory helper. The native change is confined
to that branch and preserves the existing built-in behavior.
### Test coverage
The plugin package smoke test selects devices from the registered
library, disables CPU fallback, checks inference results and provider
options, and releases sessions before unregistering the library.
Minimum-runtime package CI keeps using
`SessionOptions.add_provider_for_devices`, with its runtime floor
unchanged at 1.30.1. `ORT_WEBGPU_TEST_PROVIDER_NAME=1` additionally
tests the updated Python binding.
The Windows plugin PR job builds the current plugin host wheel. Its
existing `Run tests` step installs the locally built host and plugin
wheels, enables provider-name coverage, and requires a WebGPU device. It
reuses the job's Python environment and installs dependencies once; the
local wheel installation uses `--no-deps`. The plugin GPU job depends
only on the plugin compile job. The built-in build and test jobs retain
their original configuration.
The Windows delay-load test is limited to built-in WebGPU, matching its
use of `AppendExecutionProvider("WebGPU", ...)`. Building the plugin
host wheel also enables the ORT shared library, which previously caused
this built-in-only test to be included in plugin builds.
### Validation
- Windows x64 Release/D3D12: rebuilt and reinstalled built-in and plugin
host Python wheels using the existing build folders.
- Built-in host: inference passed with CPU fallback disabled and no
external plugin registered.
- Plugin host: registration, explicit device selection, provider-name
selection, provider options, and inference passed with CPU fallback
disabled.
- Released 1.31.0 host: default compatibility smoke passed. An exact
1.30.1 Windows wheel was unavailable from both package feeds, so it is
not claimed as a local compatibility pass.
- Integrated CI Python commands: executed locally against existing
plugin build artifacts after installing the combined job requirements.
Wheel packaging, installation with `--no-deps`, inference, and `pip
check` passed. The installed Python binding hash matched the build
artifact.
- Reproduced the plugin delay-load failure locally. Regenerated both
existing CMake build folders and verified that only this test was
removed from the plugin test list; the built-in list was unchanged.
Built and passed the built-in delay-load test, and reran the plugin
Python regression successfully.
- `lintrunner`, `git diff --check`, `actionlint` with custom runner
labels, and PowerShell syntax validation passed.
Linux, macOS, static-plugin builds, and the complete remote workflow
were not run locally. The full native suite was not rerun for the
test-selection fix. CI time savings have not been measured.
### Description <!-- Describe your changes. --> Reintegrate the ONNX 1.23 base plumbing originally introduced by #32359 and reverted by #33073, now using ONNX **1.23.2**. - Coordinate source/submodule, FetchContent, vcpkg, and Python dependency pins and raise the relevant opset ceilings to 28. - Restore Float6 public element-type mappings and conversion coverage, preserving subsequent independent type-support changes. - Update existing-type Cast registrations, CPU/CUDA Mod boundaries, signed BitShift support, provider compatibility, backend filters/tolerances, and operator documentation. - Materialize the removed upstream node-test corpus with exact-version and minimum-case guards, and cover backend serialization and released-opset policy. - Keep matching FetchContent/vcpkg compatibility patches; remove the released-opset correction now fixed upstream in 1.23.2. - Address review feedback: OpenVINO rejects BitShift-28; CPU/CUDA legacy SpaceToDepth kernels stop at 27; DirectML accepts only safe DCR; unverified graph providers defer SpaceToDepth-28. Full builds use ONNX function expansion for CRD, with literal-output and routing regressions. Avoid creating unsupported SpaceToDepth-28 nodes in the post-expansion Slice/Concat fusion. - Use shared ABI-safe tensor-element conversion in the native ONNX node runner, fixing INT2/UINT2/FLOAT8E8M0 corpus decoding after the main merge. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> This is a redo of #32359 after its rollback in #33073. ONNX 1.23.2 includes the external tensor read/re-save fix from onnx/onnx#8472 for onnx/onnx#8471 and the released-opset metadata fix from onnx/onnx#8544 for onnx/onnx#8529. The redundant local `LastReleaseVersionMap` patch has been removed. Remaining FetchContent/vcpkg patches are byte-identical and apply against the pinned release. **Scope:** This PR establishes dependency/runtime base integration. Full per-operator opset-28 support remains separate follow-up work; see #32362, #32369, #32372, #32378, #32381, and #32393. These references describe staged scope, not a claim that every follow-up has landed or every opset-28 operator is supported. Runtime function expansion is a full-build fallback, not minimal/ORT-format runtime coverage. ### Local validation Completed on Windows with retained CPU-only Ninja/MSVC `RelWithDebInfo` builds and strict released-opset policy: - Built `onnxruntime_provider_test`, `onnxruntime_test_all`, `onnx_test_runner`, and Python bindings against ONNX 1.23.2; retained build remains available for incremental work. - Source-built and installed the exact patched ONNX 1.23.2 Python wheel. - Separate `--minimal_build extended` shared runtime configured, compiled, and linked successfully with ML/contrib ops and unit tests disabled. This exercises `ONNX_MINIMAL_BUILD=ON`; it is not a minimal-runtime function-op execution test. - Mod/BitShift/SpaceToDepth/Cast provider tests: **247 passed**, no skips. SpaceToDepth checks default DCR, explicit DCR, and CRD with distinct literal expected outputs, expansion, and legacy opset boundaries. - Slice/Concat fusion and type-info/tensor-proto framework tests: **27 passed**, no skips. - Python backend tests: **22 passed**, plus **2 subtests passed**. - Full Python backend series: **4,066 tests**, **2,685 skipped**, zero failures. Skipped/unsupported coverage is not a pass. - Materialized **1,884 node cases**. Native CPU runner with `-m 1500` collected **1,760 cases**: **1,590 succeeded**, **170 not implemented**, **zero failed**, exit 0. These results do not establish complete opset-28 support. - Generated CPU operator documentation matches the rebuilt kernel registry; verified archive hashes, patch identity/application, submodule SHA, and changed-file lint. **Remaining validation/infrastructure limits:** GPU and other EP execution relies on CI; optional-provider routing checks are compiled into the relevant builds but were unavailable locally. Full unfiltered C++ suites and the ancillary broad `ALL` target were not rerun. The oracle-dependent corpus equivalence test skips because upstream no longer ships the corpus. The new ONNX 1.23.2 tarball currently returns **404 from the Microsoft vcpkg asset mirror**; it still needs Terrapin/infra seeding before read-only `x-block-origin` vcpkg legs can download it. FetchContent and the successful minimal build are mirror-independent. New-head CI is pending. --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: gramalingam <10075881+gramalingam@users.noreply.github.com>
### Description Adds transposed-B support to the KleidiAI FP16 GEMM path on SME/SME2. - Uses the KleidiAI nxk RHS packer for transposed B matrices. - Propagates the B layout through MLAS and validates leading dimensions. - Adds coverage for transposed-B execution and unsupported configurations. ### Motivation and Context Fixes a performance regression where FP16 GEMM workloads with a transposed B matrix fell back to Eigen, even on SME/SME2 systems that support an optimized KleidiAI packing path. This change allows those workloads to use accelerated MLAS Half GEMM. --------- Signed-off-by: Patryk Kaiser <patryk.kaiser@arm.com>
### Description <!-- Describe your changes. --> ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: eserscor <247253654+eserscor@users.noreply.github.com>
## Summary Fixes #33272 Corrects the CPU `OneHot` operator's handling of fractional floating-point indices to comply with the ONNX specification. Previously, valid fractional indices could silently produce all-off rows instead of selecting their corresponding class after conversion to `int64`. ## Changes - Convert finite, representable floating-point indices to `int64` before negative-index adjustment and class comparison. - Safely treat nonfinite and unrepresentable floating-point indices as out of range. - Preserve existing integer-index handling. - Add CPU regression tests for positive and negative fractional indices, out-of-range values, nonfinite values, and integer-valued floats. - Cover both opsets 9 and 11 for fractional-index behavior. ## Testing **Verified:** - Independently reproduced the original bug with ONNX Runtime 1.31.0 on Kaggle/Linux using `CPUExecutionProvider`. - Also reproduced the original behavior with ONNX Runtime 1.23.2 locally. - `clang-format` passed. - Full `lintrunner` passed. - `git diff --check` passed. **Not yet verified:** - The patched C++ implementation has not been successfully built or executed. - The new C++ regression tests have not been run. - The Kaggle source build is not yet confirmed successful. Local compilation was blocked by missing build dependencies and toolchain components. CI validation is needed to establish whether the patch compiles and passes provider tests. ## AI assistance OpenAI Codex assisted with investigation, implementation, and regression-test development. I independently reproduced the reported behavior, reviewed the source changes, and verified the available formatting and lint results. ## Related issue Fixes #33272
### Description Extended ORT pattern matching to replace 6 Slice + 1 Concat with the existing SpaceToDepth operation, reducing operator overhead. Convolution Load Balancing: Improved thread-wise distribution of convolution work to avoid uneven workloads and thread idle time, improving multi-thread performance Observation: YOLOX-Tiny gains ~16.8% at 2T and 15.4% at 4T, YOLOX-Nano gains ~8% at 2T and at 4T, while YOLOX-Small shows no difference <img width="489" height="262" alt="yolox" src="https://github.com/user-attachments/assets/e99f6cfb-74cc-4a75-bc87-d50051c147a9" /> --------- Co-authored-by: Manogna-Sree <elisetti.manognasree@multicorewareinc.com>
### Description <!-- Describe your changes. --> Include the invalid-shader variant in the program cache hint and generate the explicit build key using CalculateProgramCacheKey. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Fix nightly shader key validation check. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
### Description `convert_float_to_float16` could emit duplicate Cast node and output names when a shared float tensor fed multiple always-float inputs, such as `Resize` scales. The converted model then failed validation or ONNX Runtime loading. - **Cast reuse:** Cache Cast outputs by source tensor and target type so all consumers share one Cast. - **Regression coverage:** Add coverage for a shared `Resize` scales initializer listed in `graph.value_info`. ```python # Both Resize nodes now consume the same output: resize_1.input[2] == resize_2.input[2] == "scales_cast_to_fp32" ``` ### Motivation and Context Dynamo-based PyTorch ONNX export deduplicates identical constants and records them in `value_info`, exposing this failure for models with multiple upsample layers. <!-- START COPILOT CODING AGENT SUFFIX --> - Fixes #33088 --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com> Co-authored-by: xadupre <22452781+xadupre@users.noreply.github.com> Co-authored-by: Tianlei Wu <tlwu@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
) ### Description Adds an **opt-in, hardware-keyed on-disk tactic cache** for the fpA_intB weight-only `MatMulNBits` path, plus a reduced first-time profiling sweep with lazy fallback. Persistence is off by default: with no cache location configured, behavior is unchanged (in-process only). **Cache subsystem** (`gemm_tactic_cache.{h,cc}`) - `HardwareSignature` (GPU name, SM, CUDA runtime, ORT version; commit/build config/driver/MP count recorded for diagnostics) names and guards cache files, so tactics are never reused across incompatible GPUs/toolkits/ORT versions. A strict `tactic_selection_version` header field rejects caches tuned under an older selection policy, even when the ORT version is unchanged. - TSV format: percent-encoded free-text fields, a column-name header (appended columns are backward/forward compatible), file locking + atomic write through a per-writer temp file and rename (`MoveFileExW` on Windows; paths are UTF-8), in-memory merge on flush so concurrent sessions tuning different shapes don't lose updates, and creation of a missing cache directory on first write. - `MatMulNBitsTacticCache` keyed by `(N16b, K, act dtype, weight type, bits, block_size, zero-points, gemv_enabled, packing_sm)`. `packing_sm` keeps the SM80-compat and native-SM90 tactics for the same shape distinct on one device (e.g. H200). 2-, 4-, and 8-bit weights are keyed separately. **Profiler** (`gemm_profiler.h`, `fpA_intB_gemm_profiler.{h,cc}`) - Reduced initial sweep to a small M bucket set (`{1,2,4,…,2048}` default; override with `ep.cuda.fpa_intb_profile_m` / `ORT_FPA_INTB_PROFILE_M`). `getBestConfigOrProfile` lazily profiles any missing bucket on first use. - Persistent-cache load/store hooks are no-ops unless a cache is attached. - CUTLASS tactics loaded from disk are validated against the runner's current `getConfigs()` and dropped (re-profiled) if incompatible. **Wiring** (`matmul_nbits.{h,cc}`, `cuda_execution_provider.cc`, plugin `cuda_ep.cc`) - Cache location: session config (`ep.cuda.gemm_tactic_cache_prefix` / `_dir`) takes precedence over env (`ORT_CUDA_GEMM_TACTIC_CACHE_PREFIX` / `_DIR`); a prefix wins over a dir. - Process-global caches are keyed by resolved file path; the file is loaded once per location (not once per node), and the signature is built from the kernel's cached device properties. - The process-global in-memory profile map (`GemmIdCore`) is keyed by the same quantization fields as the persistent key (bits, block size, zero points, GEMV support), so kernels with different quantization never share tactics. - Construction-time sweeps and lazily profiled buckets are staged in memory as soon as they are profiled, and all disk writes are batched at CUDA EP teardown (`~CUDAExecutionProvider`, or `~CudaEp` for the plugin EP), so there is no disk I/O on the inference path. - Lazy profiling is skipped while the compute stream is capturing a CUDA graph; run a warmup inference before capture. **Offline tuning tool**: `onnxruntime/python/tools/fpa_intb_tune.py` (uses session config entries; releases the session before summarizing so lazily tuned buckets are flushed). ### Testing - `gemm_tactic_cache_test.cc` (built into `onnxruntime_providers_cuda_ut` with `onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON`, run via `--gtest_filter=CUDA_EP_Unittest.All`): TSV round-trip and malformed escapes, SM80/SM90 config serialize/parse, null tactic, store/load, signature-mismatch rejection, non-strict commit/build config, selection-version rejection, appended-column tolerance, concurrent flush-merge, missing-directory creation, file path resolution, and `GemmIdCore` separation of quantization variants. - `*MatMulNBits*` CUDA integration tests. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
### Description Native inference can finish successfully, write its profiling output, and then hang indefinitely while Windows terminates the process. Skip 1DS uploader teardown in `OneDsTelemetry::~OneDsTelemetry` when Windows DLL shutdown is in progress, before acquiring telemetry locks. The Windows helper resolves `RtlDllShutdownInProgress` from the already loaded `ntdll.dll`. This process-wide check also works for telemetry compiled into an EP plugin DLL. Normal object destruction and explicit `FreeLibrary` retain the existing cleanup path; non-Windows behavior is unchanged. Pending telemetry is not flushed during process termination. ### Motivation and Context Two captured native benchmark hangs had only the terminating thread remaining, with this shutdown stack: ```text ExitProcess -> LdrShutdownProcess -> WindowsEnv static destructor -> OneDsTelemetry::~OneDsTelemetry -> LogManagerImpl::FlushAndTeardown -> HttpClientManager::cancelAllRequests / cancelAllRequestsAsync -> HttpClient_WinHttp::CancelAllRequests -> condition-variable wait ``` Windows terminates the other threads before calling DLL process-detach destructors. The uploader can therefore wait forever for WinHTTP callbacks that can no longer run; a telemetry lock may also be held by a terminated thread. Checking shutdown state before taking locks avoids both waits. Resolved ORT/SDK frames were verified against identical machine code in a build with symbols. ### Regression Coverage Three committed Windows CTest cases load a fixture DLL linked to the production `onnxruntime_common` telemetry implementation. A test peer injects a mock log manager with network telemetry disabled; no model, GPU, or symbol-address patching is required. - `onnxruntime_telemetry_process_exit_test`: calls `ExitProcess` with a pending mock callback and verifies that the real DLL-owned telemetry destructor skips uploader cleanup. - `onnxruntime_telemetry_normal_destruction_test`: destroys the object before DLL unload and verifies one `Flush`, one `FlushAndTeardown`, and cleared shared state. - `onnxruntime_telemetry_explicit_unload_test`: verifies the same cleanup during `FreeLibrary`. The parent requires both a successful child exit and an inherited event signaled only after teardown assertions pass. Each child has a 10-second timeout, with a 15-second CTest limit. The mock drain itself is bounded to two seconds so a regressed shutdown cannot strand a child inside DLL teardown. Run from the CMake build directory after building `onnxruntime_telemetry_shutdown_test`: ```text ctest -C Release -R "^onnxruntime_telemetry_(process_exit|normal_destruction|explicit_unload)_test$" --output-on-failure ``` ### Validation - [x] Windows x64/MSVC Release: configured a separate build and compiled the committed test target and its production dependencies; all three lifecycle tests passed. - [x] Local mutation checks: removing the shutdown guard failed only the process-exit case; returning unconditionally from the destructor failed both normal destruction and explicit unload. Restoring the fix passed all three cases again. - [x] Windows/MSVC runtime validation: recompiled the changed translation units and relinked isolated built-in WebGPU ORT, plugin ORT host, and WebGPU plugin binaries using compatible cached build objects. - [x] Original local lifecycle harness with network telemetry suppressed and a mock pending uploader injected into the actual ORT DLL: baseline entered SDK draining and timed out; fixed runtime exited normally in **20/20** trials. Explicit DLL unload with an inactive uploader passed **3/3** trials for both baseline and fixed runtimes. - [x] Native GenAI benchmark: **built-in/plugin WebGPU × R32 FP16 LoRA on/off**, 1,024 prompt tokens and 32 output tokens, batch 1, one warmup and two measured runs. All four configurations exited with code 0; generated text matched across providers for each adapter state. An additional built-in/LoRA-on run matched the original runtime's output and exited normally. - [x] Changed-file `lintrunner -a` and `git diff --check` passed. The committed lifecycle tests use a mock uploader; they do not cover real pending-upload unload stress. The benchmark executable, GenAI DLL, model graph, configuration, and adapter were unchanged during runtime validation.
### Description <!-- Describe your changes. --> - Use a packaged LICENSE file and set the project URL in CUDA NuGet metadata. - Stage the repository LICENSE and ThirdPartyNotices.txt alongside the project before packing. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Aligns CUDA plugin EP NuGet packaging with the WebGPU package updates in #28525. The generated package was verified to contain the license files and expected NuGet metadata; packing succeeded with a temporary linux-x64 fixture. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
### Description Add the missing safety checks in two training loss graph optimizers: - `CastSceLossFusion` now skips a `SoftmaxCrossEntropyLossInternal` input with no producer instead of dereferencing a null node. - `SoftmaxCrossEntropyLossInternalFusion` only removes `LogSoftmax` when its normalized axis is 1 and its result is neither public nor shared. The optional Cast path receives the same use check. ### Motivation and Context The previous code could segfault on a valid graph input, change loss values for another LogSoftmax axis, or remove a declared graph output. Fixes #32978 Fixes #32979 ### Testing Built `onnxruntime_test_all` with training enabled in Release mode. Five focused tests passed, including the new graph-input, axis, and public-output regressions and the two existing successful loss-fusion paths. Also rebuilt the CUDA Training Python extension and replayed the public reproducers on an RTX 5080 with CPU fallback disabled: - direct graph-input scores optimize and run without a crash; - axis 0 and public `log_prob` keep the original `LogSoftmax -> NegativeLogLikelihoodLossInternal` graph and outputs; - the axis 1 control still fuses and preserves its output. --------- Signed-off-by: Yuhx141 <155034285+Yuhx141@users.noreply.github.com>
Reverts #29789 As discussed with the PR author in [this comment](#29788 (comment)), this PR is not necessary. We should revert it to avoid any misunderstandings regarding constant folding. --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 80e16013-a359-4acd-b6ca-583ffdd4f313
## Summary Add WebGPU support for `com.microsoft::MatMulBlockQuantizedFp8Weight`, the long-named dense projection operator used by the FP8 model export. This remains necessary independently of MoE/QMoE support and targets `main` directly. - Read row-major E4M3FN weights with FP32 block scales `[N, ceil(K/block_size)]`. - Support optional activation FP8 quantize/dequantize, bias, rank-one/batched inputs, partial blocks, non-word-aligned rows, and empty outputs/reductions. - Support FP16 activations. BF16 storage is not supported by this WebGPU kernel. - Use FP16 subgroup-matrix operands with FP32 accumulation for supported prefill configurations, with a scalar FP32-reduction fallback. This is not native FP8 matrix hardware compute. - Preserve NaNs during activation quantization and use explicit E4M3 ties-to-even rounding. - Preserve the FP16-only operator during cast insertion and prefer a registered WebGPU plugin over a built-in factory when applicable. ## October 16 refresh and review fixes Merged `main` at `e88538cc6f` and pushed the review fixes in `4905cd01f2`. - Classify activation NaNs through their bits before saturation. A floating-point self-comparison was optimized away on the real Vulkan backend; the bitwise check passes the added NaN regression. - Use unequal positive/negative subnormal weights, so flushing both values to zero no longer leaves the reference unchanged. - Make the shared `subgroup_id` builtin opt-in per program and check the separate WGSL language feature. Ordinary shaders and scalar fallbacks no longer acquire that requirement. - Gate both dense and routed block-FP8 matrix paths, which share the builtin, on the language capability. Include the opt-in in the program cache key. - Add capability/cache-key regression coverage and document the operator and fallback behavior. ## Current-head validation - Linux Release static WebGPU provider-test and general unit-test builds passed. - All **16 dense FP8 tests** passed on H200/Vulkan with CPU EP fallback disabled. Matrix-specific cases ran rather than skipped, including distinct subgroup tiles and a second workgroup. - All **38 related WebGPU QMoE tests** passed, covering the shared shader-generation change. - All **four focused cast-transformer tests** passed. - Incremental repository-standard lintrunner and `git diff --check` passed. - Fresh remote CI is running. Browser/Metal portability and macOS arm64 results must be confirmed by the new CI runs; local Vulkan results alone do not establish those. ## Earlier evidence and limits An earlier shared-plugin build ran builder-exported W8A16 and W8A8 dense projection nodes with CPU fallback disabled and matched their CPU references. Earlier synthetic prefill trials showed the tiled path faster than its forced scalar control, but these are historical measurements, not a new performance claim for this refreshed head. This PR's focused validation is not a full-model quality or throughput qualification. Routed FP8 expert execution and native-FP8 WebGPU/Dawn proposal work remain separate scopes. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
### Description Conv2dMM assembles vec3 inputs from scalar storage but divided the channel coordinate by four. This read the wrong channels for input counts such as 6, 9, and 15. Use the scalar storage coordinate for this path. Add FP32/FP16 regression tests with dynamic and initializer weights, plus 3-channel, scalar, and vec4 controls. ### Motivation and Context Fixes #32928. The attached nine-channel model still reproduces the reported error on current main. Three-channel inputs hide the bug because their starting channel is always zero. ### Validation - Windows/D3D12 Release build and lint pass. - `onnxruntime_provider_test --gtest_filter=*Conv2dMMInputChannelsTest*`: 28 passed, no skips; 12 cases failed before the fix. - Attachment maximum error versus CPU drops from 0.6985353082 to 1.788e-7. Profiling confirms WebGPU Conv with CPU fallback disabled. - macOS Metal was not tested.
…33323) ## Stack dependency Current stack: `main` -> **#33323** -> #33324. #33094 and #33328 have merged. This PR now targets `main`; their packing, K-lane tuning and expert-reuse changes are upstream dependencies, not work claimed by this PR. ## Changes - Add default-off `ORT_NVFP4_GEMV_NATIVE_SCALE` to use the existing CUDA native scalar E4M3 converter for block scales in offline K-packed NVFP4 GEMV. - Retain the dependency's K-lane selection (four for K <= 1024, eight otherwise), expert-row reuse, packing contract, FP4 decoding and accumulation. - Keep software conversion on other compiled architectures and leave N-packed GEMV and GEMM/prefill unchanged. - Add an incremental all-256-code numerical regression for FP16/BF16 outputs, including NaN handling and both K-lane configurations, plus behavior documentation. This is scalar byte conversion, not native FP8 matrix multiplication. Set the flag before first use; dispatch specialization caches it. ## Measured performance: original optimization stack This optimization improved full-model throughput in a controlled flag-off/flag-on comparison of the original implementation: - E2E throughput: **104.9262 -> 108.0584 tokens/s (+2.99%)**. - Decode throughput: **133.4966 -> 138.5600 tokens/s (+3.79%)**. - Control: `ORT_NVFP4_GEMV_NATIVE_SCALE=0`; candidate: `ORT_NVFP4_GEMV_NATIVE_SCALE=1`. Measurement used Qwen3.8 Flash Next's NVFP4 Engine package with FP8 MTP on one otherwise idle NVIDIA H200: batch one, **8,192 uncached input tokens, 512 output tokens, greedy generation, MTP draft width two**. Each variant pooled ten measured requests across counterbalanced invocations, with one excluded warmup per invocation and no discarded measured requests. Loading and warmup are excluded. Python 3.12.15, CUDA compiler 13.3.73, native SM90 compilation and driver 580.105.08 were matched between variants. Both variants used the same sparse-attention optimizations and provider binary; only the native-scale flag differed. A subsequently rejected wider-K experiment was present in that historical binary but disabled in both variants. E2E TPS is `512 * request_count / sum(request_seconds)`; decode TPS is `511 * request_count / sum(decode_seconds)`. Both ten-request pools passed the benchmark validator. **These are measured gains from the earlier combined optimization stack, not measurements of this refreshed PR head or an isolated stock-main baseline.** The published port preserves the parent PR's dispatch and has passed the correctness checks below, but its end-to-end performance must still be remeasured on the final integrated ORT/GenAI stack. No gain is claimed for default-off operation. ## October 16 review updates The NVFP4 regression now makes every K block contribute through both packed weight nibbles, with varying scales and a full-reduction CPU reference. K=80 exercises all four K lanes plus a later iteration; K=1040 exercises all eight lanes plus later iterations. - Added checked-in CTest runs for both flag values in separate processes; no in-process environment toggling. - CUDA graph-node inspection checks the actual native/software template specialization selected by the launcher. Native-specific tests also verify FP8 build support and the loaded kernel's binary/PTX compilation targets; unsupported native coverage skips explicitly. - All four new mode-specific CTest processes passed on the combined stack: six inner test executions passed and two native-only cases intentionally skipped in flag-off processes. Both flag-on native-specific cases ran and passed, not skipped. - The changed CUDA translation units and regression source compiled, and the CUDA internal-test module relinked successfully against the existing build. This was targeted incremental validation, not a new clean full build. - Standard incremental lintrunner and whitespace checks passed. Both regression sources also passed FP4-enabled and FP4-disabled compilation-only checks with warnings treated as errors. - No throughput benchmarks or new performance claims. Fresh remote CI is pending. ## Earlier validation before the October 16 review updates - Clean CUDA provider and internal-test runner build using CUDA 13.3.73 on H200. - Four current internal QMoE CUDA tests passed in each of four fresh-process flag combinations: NVFP4 native scale off/on x FP8 native decode off/on (16 test executions, with nonzero inner counts verified). - Tests were run on the combined child stack: two dependency compaction/dequantization tests and our two all-code GEMV regressions. The FP8 child covers actual CUDA graph capture/replay. Two tests removed from the dependency were not restored or counted. - `clang-format` 20.1.8 dry-run with warnings treated as errors and `git diff --check` passed on both incremental heads. - Both regression files also pass compilation-only checks with FP4 build gates disabled and `-Werror`. - The previously failing `FpAIntBGemvTest.TacticCacheSeparatesPairedAndWaveAwareModes` and `MatMulNBitsWorkspace.CompactEligibilityMatchesRcContract` both pass on the refreshed stack. ## Ownership and scope This PR does not import or claim collaborator-owned offline packing, K-lane tuning, expert reuse, scale/address hoisting, sparse attention, allocator changes or packed prefill. They remain separate dependencies/work. --------- Co-authored-by: Akshay Sonawane <asonawane@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Kunal Vaishnavi <kvaishnavi@microsoft.com>
> **Stacked on #33094.** The base branch is `kvaishnavi/qwen38-nvfp4-offline-packing`, so the diff below is this change only. Once #33094 merges, this PR will be retargeted to `main`. Opened as a draft until then. ## Description Adds `MoeGemvFp4DecodeKernel`, a faster decode kernel for the offline K-packed (`weights_prepacked=1`, row-major `[E, N, K/2]`) NVFP4 QMoE GEMV that #33094 introduced. It reproduces `MoeGemvFp4RawKPackedKernel`'s fp32 summation order **bit-exactly**, so it is a drop-in replacement: QMoE outputs are bitwise identical with or without it. On H200 it makes the Qwen3.8 FC1+FC2 decode launches 1.47x faster (BF16, 10 expanded rows) and the QMoE node 11% (1 token) to 17% (2 tokens) faster end to end. ## Summary of Changes ### Kernel | File | Change | |------|--------| | `onnxruntime/contrib_ops/cuda/llm/moe_gemm/moe_gemv_fp4.cu` | New `MoeGemvFp4DecodeKernel` and its launcher dispatch inside `LaunchMoeGemvFp4RawNPacked` (`weights_row_major` branch); the launcher now receives `sm` | | `docs/contrib_ops/cuda/moe_qmoe.md` | New "Decode kernel for offline K-packed storage" section and `ORT_FP4_GEMV_DECODE_KERNEL` table row | | `onnxruntime/test/python/transformers/test_qmoe_nvfp4_cuda.py` | New bit-exactness test (see Testing) | How it matches `MoeGemvFp4RawKPackedKernel`: - Same K-lane count (`KLanes = 4` for `K <= 1024`, else `8`, copied from #33094's launcher), same lane assignment (`lane = block(16) % KLanes`), same separate even/odd fp32 accumulators per lane, same `acc_even + acc_odd` followed by the xor tree `KLanes/2 .. 1`, same `StoreMoeGemvFp4Pair` epilogue (bias, fused SwiGLU). - One thread owns two adjacent columns and the lane pair `(2j, 2j+1)`: one 128-bit load covers two blocks, the xor-tree steps at lane distance `KLanes/2 .. 2` are shuffles between the `KLanes/2` threads of a column, and the last step (distance 1) is an in-thread add. - Speed comes from 128-bit streaming weight loads with 2 K tiles in flight, table-free E2M1 decode (`fp4 * 2^-126` bf16 / `fp4 * 2^-14` fp16 bit patterns times a rescaled block scale, a single rounding identical to `T(fp4 * scale)`), E4M3 scale conversion with `cvt.rn.f16x2.e4m3x2`, and the activation row staged once per CTA as fp32 in shared memory. Dispatch (everything else falls back to `MoeGemvFp4RawKPackedKernel` unchanged): - `weights_row_major`, `sm >= 90`, FP16 or BF16, at most 80 expanded rows, `K % 32 == 0`, `K <= 8192`, even `N`, 16/2/16-byte aligned weights/scales/activations. - Steps aside when `ORT_NVFP4_FC1_EXPERT_REUSE=1` selects the two-row reuse variant. - Opt-out: `ORT_FP4_GEMV_DECODE_KERNEL=0` (latched once per process, same convention as `ORT_FP4_GEMV_DEFAULT_TILING`). ## Testing Standalone harness (links `moe_gemv_fp4.cu` directly, calls the public launchers vs. `MoeGemvFp4RawKPackedKernel` launched exactly like #33094's launcher), H200, CUDA 13.0: - **Bit-exact: 374/374 cases, bf16 and fp16, 0 mismatching elements.** Real Qwen3.8 FC1/FC2 weights, random weights, bias on/off, fused SwiGLU on/off, invalid expert ids (`-1`, `E`), offsets-only expert map, rows 1..80 (every row count), K = 64 .. 8192 including K not a multiple of 1024 (partial last tile: 98 and 34 blocks), big/tiny/zero global scales (generic decode path), and shapes the kernel declines (fallback). Kernel timing (us/launch, CUDA-graph average over 300 launches, 512-expert L2-cold layout, real FC1 `N=1280 K=2560` fused SwiGLU and FC2 `N=2560 K=640` weights, H200): | dtype / rows | #33094 FC1 | #33094 FC2 | this PR FC1 | this PR FC2 | FC1+FC2 speedup | |---|---:|---:|---:|---:|---:| | bf16, top-10 | 18.51 | 10.27 | 12.13 | 7.49 | **1.47x** | | bf16, top-20 | 36.05 | 17.23 | 20.79 | 12.20 | **1.62x** | | fp16, top-10 | 17.55 | 9.44 | 13.77 | 8.41 | **1.22x** | | fp16, top-20 | 34.01 | 16.80 | 24.71 | 14.28 | **1.30x** | ORT-level QMoE node (Qwen3.8 shape `hidden=2560 inter=640 top_k=10`, 512 experts, BF16, `weights_prepacked=1`, routing rotated over 64 inputs; GEMV kernel time per node from CUPTI, wall time per `run_with_iobinding`): | tokens | GEMV us/node #33094 | GEMV us/node this PR | node wall us #33094 | node wall us this PR | |---:|---:|---:|---:|---:| | 1 | 28.97 | 19.44 | 88.5 | 78.5 | | 2 | 52.18 | 33.24 | 113.3 | 94.5 | The node output is bitwise identical with `ORT_FP4_GEMV_DECODE_KERNEL=0` and `=1` in this benchmark too. Python tests, CUDA plugin build of this branch (sm_90 only, H200, `CUDA_VISIBLE_DEVICES=2`, `ORT_TEST_CUDA_PLUGIN_EP=1`): - New: `test_nvfp4_offline_row_major_decode_gemv_bit_exact` (FP16 and BF16). Runs five `weights_prepacked=1` QMoE cases (Qwen3.8 shape at 1/2/6 tokens, i.e. 10/20/60 expanded rows, with and without bias; a partial-K-tile shape; a `K % 32 != 0` shape that the decode kernel declines), each checked against the PyTorch reference, and requires the output to be bitwise equal to a fresh process with `ORT_FP4_GEMV_DECODE_KERNEL=0`. It also uses `torch.profiler` to assert that the decode kernel runs in the default process and `MoeGemvFp4RawKPackedKernel` runs with the opt-out. - Full `python -m pytest onnxruntime/test/python/transformers/test_qmoe_nvfp4_cuda.py`: 178 passed, 8 skipped (opt-in large-memory tests), 0 failed (including all `offline_row_major` and `kpacked` tests from #33094). - `lintrunner` (CLANGFORMAT, RUFF, RUFF-FORMAT) on the touched files: no issues. Known limitations: - The table-free decode needs every block scale below 4 (`|global_scale| * 448 < 3.98`; Qwen3.8 has at most 0.09). A per-expert fallback to a generic `Fp4I2FConverter` decode keeps results bit-identical for larger scales, but it is about 30% slower than `MoeGemvFp4RawKPackedKernel` (BF16 top-10: 24.0/13.3 us vs 18.6/10.3 us). Such checkpoints should set `ORT_FP4_GEMV_DECODE_KERNEL=0`. - FP16 gains less than BF16 (1.22x vs 1.47x at top-10). ## Motivation and Context #33094 added the K-packed row-major layout and its native `MoeGemvFp4RawKPackedKernel`. At Qwen3.8 decode shapes that kernel issues 8-byte weight loads and re-reads/decodes the activation row per 16-element block. This PR keeps its numerics and replaces only the memory/issue pattern. Related: #33323 / #33324 touch the same launcher (opt-in native E4M3 scale decoding) and are based on an earlier #33094 head, so they need a rebase onto #33094 regardless of this PR. The two are semantically compatible: the native-scale option only changes how E4M3 scales are converted (exactly), and the decode kernel already uses the hardware E4M3 conversion. The only overlap is a textual conflict next to the launcher dispatch (#33323 adds a parameter to the `launch_k_packed` lambda right below the new decode dispatch). ## Checklist - [x] Tests added/updated - [x] No breaking changes (default behavior is bitwise identical; `ORT_FP4_GEMV_DECODE_KERNEL=0` restores the previous kernel) - [x] Documentation updated (`docs/contrib_ops/cuda/moe_qmoe.md`) --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…3392) ### Description Skip `MatmulBNFusion` unless the Reshape and Transpose nodes between MatMul and BatchNormalization keep the MatMul output columns on the BatchNormalization channel axis. The check follows the column axis through each node's inferred output shape, and skips the fusion if a shape is unknown. Add a test for the Reshape from #33364 and for a Reshape that keeps the channels, comparing against the unfused graph. `FuseMatmulBNWithOnlyReshape` and `FuseMatmulBNWithOnlyTranspose` now expect no fusion. Their models fail without optimizations (`Invalid input scale: 0th dimension != 1`), since the channel axis has size 1 and scale has 5 values. ### Motivation and Context The fusion scales column j of the MatMul weight by channel j of the BatchNormalization. A Reshape such as [2, 4] -> [1, 4, 2, 1] moves the columns off the channel axis, so the Basic, Extended and All levels returned wrong values. Fixes #33364. ### Validation Linux aarch64 CPU, Release build: - The new test fails before the change and passes after it. - `GraphTransformationTests.*`: 430 passed, 18 skipped (17 CUDA/WebGPU-only, 1 FP16 fallback test skipped on this CPU). - The repro from #33364 matches the expected output at every optimization level. - clang-format 20.1.8 and `git diff --check` pass.
… M <= 4 (#33299) ## Description The fused INT4 GEMV and small-M batched CUDA kernels in `MatMulNBits` require `N % 8 == 0`, so projections such as `N=324` (e.g. hyperconnection projections in Qwen3.8 Flash-Next) fall back to dequantize + GEMM. This adds an **opt-in, default-off** selector, `ORT_MATMULNBITS_UNALIGNED4_FAST_PATH=1`, that reuses the existing aligned kernels with masked tail scales and warp-uniform output guards for `N % 8 == 4`, `1 <= M <= 4`. Valid outputs are bit-identical to the original aligned kernel evaluated on the same weights padded to the next multiple of 8 columns (it is **not** bit-identical to the dequantize + GEMM fallback, whose arithmetic differs). Default behavior, aligned shapes and every ineligible configuration are unchanged. ## Summary of Changes | File | Change | |------|--------| | `matmul_nbits_unaligned4.h` (new) | Strict `0/1` env parsing, eligibility policy (`kUnaligned4MaxM = 4`), once-per-branch dispatch receipt. | | `matmul_4bits_m1_impl.cuh`, `matmul_4bits_half.cu` | `TailN` variant of the M=1 shared-scale kernel: last CTA loads only valid scale rows (zero fill); all warps join the barrier before invalid output warps exit without reading weights or writing output. | | `matmul_4bits_batched_impl.cuh`, `matmul_4bits_batched.cuh`, `matmul_4bits_batched_half.cu` | `TailN` variant of the register-batched kernel: invalid warps exit before any tensor read (no CTA barrier in this kernel). | | `matmul_nbits.{h,cc,cuh}` | Read the selector at kernel construction and dispatch only for FP16, `bits=4`, `block_size=32`, column-wise, no zero-points/bias/`g_idx`, no fpA_intB, no forced chunking. | | `docs/contrib_ops/cuda/matmul_nbits.md` | New §4.5: contract, receipts, measurements, validation. | | `test/python/transformers/test_matmul_nbits_unaligned4.py` | Paired byte-exact test against the padded aligned kernels, plus ineligible-config equivalence. | Eligibility stops at `M = 4` deliberately: the kernels support `M <= 8`, but for `M = 5..8` the batched tail kernel measured slower than the fallback. ## Testing Built from this tree's parent base `e379df24db` + this change (CUDA 13.0, SM90, `onnxruntime_BUILD_CUDA_EP_AS_PLUGIN=ON`); upstream has since moved 24 commits without touching these files, and the branch is rebased on `988cc07aed` without re-running the GPU tests. - `python -m unittest onnxruntime/test/python/transformers/test_matmul_nbits_unaligned4.py -v` (run from `transformers/`): **144 paired byte comparisons** (M 1-4 × N 4/12/324 × K 32/320/10240 × constant/signed-zero/adversarial/realistic data, with poisoned padded rows) pass with both the CUDA EP and `ORT_TEST_CUDA_PLUGIN_EP=1`; M=5, M=8, M=9, bias, zero-point and `g_idx` produce identical bytes with the selector on and off. - `compute-sanitizer` memcheck and synccheck over the same test: 0 errors. (Allocations come from ORT's arena, so this is not an exact-boundary OOB proof.) - Existing `quantization/test_op_matmul_4bits.py` and `test_op_matmulnbits_prepacked_cuda.py`: 27 passed, 11 skipped. - Not run: C++ `onnxruntime_provider_test` (unit tests were disabled in this build), non-SM90 GPUs, Windows. Operator-level timing on one H200 (median of 7×500 `Run`s with device-resident I/O; includes launch overhead), fallback → selector, µs: | N × K | M=1 | M=2 | M=4 | M=5 (not dispatched) | | --- | --- | --- | --- | --- | | 324 × 10240 | 24.7 → 19.3 | 28.9 → 17.7 | 26.3 → 20.7 | 29.0 → 29.1 | | 324 × 4096 | 21.7 → 16.4 | 25.6 → 16.0 | 23.6 → 17.3 | – | | 4100 × 2048 | 26.9 → 16.1 | 30.8 → 17.1 | 31.0 → 19.0 | – | In a private end-to-end Qwen3.8 Flash-Next 2K decode experiment (not reproducible from this PR) the selector contributed to a +12.7% decode speedup; that run used other private changes and `M <= 8`, so no end-to-end claim is made here. ## Motivation and Context Keeping the selector opt-in avoids changing numerics for existing users, since the result differs bitwise from the dequantize + GEMM fallback. Promoting it to default would need model-level quality validation. ## Checklist - [x] Tests added/updated - [x] No breaking changes (default off; ineligible shapes unchanged) - [x] Documentation updated --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Part of the grouped generic CUDA operator stack. Preserve the qualified source snapshot.
This was referenced Oct 19, 2026
Akshay Sonawane (apsonawane)
added this pull request to stack #33466
October 19, 2026 18:03
#33120) ### Description Make `onnxruntime_DISABLE_EXCEPTIONS` / `--disable_exceptions` usable without `onnxruntime_MINIMAL_BUILD` / `--minimal_build`, and build a regular (non-minimal) CPU configuration with exceptions disabled in CI. - `onnxruntime_DISABLE_EXCEPTIONS` stays a `cmake_dependent_option` gated on `NOT onnxruntime_ENABLE_PYTHON`, but its default is now the value of `onnxruntime_MINIMAL_BUILD`, so minimal builds that rely on it defaulting to ON are unchanged. - The `FATAL_ERROR` in `adjust_global_compile_flags.cmake` and the matching `build_args.py` check are removed; `test_build_args.py` gains a `--disable_exceptions` case without `--minimal_build`. - WebAssembly keeps the old requirement: `--build_wasm --disable_exceptions` still needs `--minimal_build`, because the Emscripten triplets derive the dependencies' exception setting from the minimal build flag rather than from `--disable_exceptions`. - `-fno-unwind-tables -fno-asynchronous-unwind-tables` are now only added for minimal builds. Those flags are a binary-size optimization for mobile; a regular build with exceptions disabled keeps its unwind tables so crash reporters and profilers can still walk its stacks. - The vcpkg triplet generators in `vcpkg_helpers.py` no longer skip no-exception, non-minimal combinations (a `--use_vcpkg --disable_exceptions` regular build had no triplet to use), and apply the same unwind-table rule to dependencies; `test_vcpkg_helpers.py` covers both. ONNX's own `ONNX_DISABLE_EXCEPTIONS` handling adds the unwind-table flags unconditionally, so `cmake/patches/onnx/onnx.patch` and its vcpkg copy `binskim.patch` now add them only with `ONNX_MINIMAL_BUILD`. - model_package maps `std::system_error` to `MODEL_PACKAGE_ERR_IO` with function-try-blocks (added in #32790). They now use `MODEL_PACKAGE_TRY` / `MODEL_PACKAGE_CATCH`, which expand to nothing when exceptions are disabled, and the remaining throwing `std::filesystem` calls use their `error_code` overloads so filesystem errors still come back as a status. On Windows, path encoding conversions (`fs::u8path` on a non-UTF-8 argument, `generic_string()` on a name the code page cannot represent) still throw and have no `error_code` overload, so in that configuration they abort like other exceptions. A failed directory iteration while opening, hashing or copying is now reported as `MODEL_PACKAGE_ERR_IO` instead of silently ending the walk (a failed `increment(ec)` resets the iterator to end), so an unreadable `shared_assets` directory is an error rather than ignored. Non-minimal builds link model_package, so it has to build without exceptions for this configuration. - `DumpTensorToFile` (built with `onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS`) now uses `ORT_TRY` / `ORT_CATCH` / `ORT_RETHROW` instead of a raw `try`/`catch`, so that option also builds with exceptions disabled. - With exceptions disabled, errors ONNX and ORT would otherwise throw (for example a shape-inference failure on an invalid model) abort the process, as `ORT_THROW` already does in minimal builds. - `linux_ci.yml` gains `build-linux-x64-no-exceptions`, a MinSizeRel regular build with `--disable_exceptions`. Unit tests are not built in that configuration: 22 test sources in `onnxruntime_test_all`, `onnxruntime_provider_test` and `onnxruntime_mlas_test` use `try`/`catch` or `gsl::narrow` unguarded. ### Motivation and Context Firefox ships a regular CPU build of ONNX Runtime with exceptions disabled and has been carrying this change as a local patch. The remaining exception-dependent code in the core non-minimal build was converted in #32591, #32597 and #32883. ONNX 1.23.2 (#33134) brought in onnx/onnx#8220, which removed the last raw `throw` in ONNX's `onnx/common/tensor.h`. With this PR, current main builds locally with GCC 15 (MinSizeRel, CPU EP) with `--disable_exceptions`, both without vcpkg as static libraries and with the flags the new job's build action adds (`--use_vcpkg --build_shared_lib`).
Akshay Sonawane (apsonawane)
marked this pull request as ready for review
October 19, 2026 18:11
Contributor
Author
|
Closing this draft to replace the feature-based stack with focused branches rebuilt from current main. Existing branches are preserved; replacement PRs will be linked here. |
Contributor
Author
|
Replacement opened directly against current main: #33469. The old branch is preserved; family documentation is bundled with the replacement. |
Akshay Sonawane (apsonawane)
removed this pull request from stack #33466
October 19, 2026 19:58
Akshay Sonawane (apsonawane)
added a commit
that referenced
this pull request
Oct 24, 2026
## Summary - Add opt-in explicit FP32 arithmetic and packed input slicing for BranchwiseRMSNorm. - Add versioned RotaryEmbedding exact arithmetic while retaining existing default dispatch. - Keep RMS-chain regression coverage with this PR, independent of tensor fusions. ## Review follow-up - Honor deterministic CUDA sessions by using the existing fixed-order column reduction instead of the single-row floating-point atomic reduction. Default dispatch remains unchanged. The schema documents that deterministic summation may round differently from the default path or deterministic ReduceMean. - Add CUDA exact-mode branch-scale parity coverage for distinct `(C, H)` and flattened `(C * H)` scales over multiple outer rows, covering FP32, FP16, and BF16 input and scale storage. - Add a width-8192 single-row deterministic-session regression with a numerical reference, repeated bitwise comparison, and widened-output checks. - Prevent GroupQueryAttentionFusion from removing either RotaryEmbedding node when `use_fp32_arithmetic` is requested. Add a version-2 graph-transformer regression for query-only, key-only, both-exact, and ordinary fusion. - Regenerate the affected contrib and kernel documentation using the official renderers with source-derived metadata, including version-2 availability/history, optional widened outputs, type constraints, and CPU/CUDA/DML registrations. Full native `--gen_doc validate` remains a CI check. ## Clean main-based replacement Supersedes #33459. Reconstructed directly on `main` (`28225b8583474cf6a6e7f97222227ae668f80e8b`), without the old divergent feature-branch history. This PR targets `main` directly and has no dependency on another operator PR. Family documentation is bundled here; the separate old documentation PR #33465 is retired. ## Validation - Used the `qwen-flash` Conda environment. - Ran `lintrunner -a` before committing, followed by a passing check-only run on the full PR delta and `git diff --check`. - Per request, the review follow-up does **not** rebuild ORT/GenAI or execute the new native/GPU tests. Their execution and full generated-documentation validation are left to CI. - The original branch was previously checked with schema syntax checks and affected CUDA/host/test translation-unit compilation (SM90); this historical compile evidence does not validate the new review follow-up. - The previous head's React Native Android failure was an npm `ETIMEDOUT` downloading `supports-color`, not a source compilation failure. CUDA builds succeeded, but CUDA test jobs were canceled. The new push starts fresh CI; no claim that all pipelines are green. - Full main-based model/GPU requalification remains outstanding; keeping this PR in draft. - No benchmark/profiling/evaluation scripts or local task metadata are included.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Main integration update
Merged repository
mainat9a662b9395c7c5dc619ae4d050f2c9362d88eae3. Published head:26cb2368b5c5c7d61adef4b17b40df8414869af8. Existing PR bases remain unchanged.qwen-flashconda environment; ranlintrunner -abefore each new commit, followed by clean lint checks. Final branch-vs-main lint passes.This update supersedes earlier notes saying main has not been integrated, but does not remove the draft/build/runtime qualification requirements.
Summary
Add opt-in exact arithmetic and packed input/output support for BranchwiseRMSNorm and RotaryEmbedding, including shared reduction support, version registration, CPU/WebGPU guards and tests.
Stack and main target
Depends on #33393.
This is layer 1/7 of the microsoft/onnxruntime stack. The current review base is
asonawane/qwen-cuda-kernel-optimizationsso this PR shows only its incremental group. The final destination is upstreammain: land/reconcile prerequisites first, then rebase and retarget this layer tomain. Keep this PR in draft until that base-specific validation succeeds. Do not merge the entire stack through an unvalidated intermediate base.Cross-repository dependency: GenAI package/runtime integration requires the companion ORT contrib operators.
Validation and blockers
GatedDeltaNetTest.RecurrentOutputsIncludeFirstRowStatefails in the existing combined workspace.RejectsWrongOutputCount,QsaMalformedStateLengthsAreClamped,CsaMalformedStateLengthsAreClamped.Scope
Stack navigation
Companion repository stack: