Repository navigation
Mark cross-CPU comparisons incompatible #520
Description
Two CodSpeed Simulation reports compared GitHub-hosted runs from different CPU models and produced blocking regressions.
The test report compares oci_blob_serve_fixed_hash at 211.497 µs on an AMD EPYC 7763 with 243.455 µs on an AMD EPYC 9V74, a 13.13% regression. The commit modifies one file: crates/peryx-upstream/tests/unit/client/exec_tests.rs. We rebuilt base 6fac09c and head c6d76fcf with Rust 1.96.0 under identical build inputs and fixed source and target paths. Both benchmark executables are 644,216,480 bytes and have SHA-256 e1a9a8bd47ff4a6bab1ae30ca18e7ecb384fc73ec49df824d9eb7169cf41a74f; cmp confirms byte identity. Their benchmark and harness symbols and instructions match.
The range-unit report compares the same 211.497 µs EPYC 7763 baseline with 485.516 µs on an EPYC 9V74. That change had a separate reachable code-generation regression, which we found and fixed. The report still compares the same mismatched CPU models and flag sets. This case shows why CodSpeed should retain the report for diagnosis instead of treating every mismatch as harmless.
CodSpeed's benchmark variance documentation says shared CI runners do not guarantee the same physical machine and identifies hardware and linked libraries as causes of Simulation variance. The environment-diff announcement says recent runners collect and compare this metadata. Both reports contain a runner diff for the CPU model and flags.
Please mark a comparison incompatible or N/A when the collected metadata proves that the base and head CPU model or flags differ. Retain the report and environment diff for diagnosis while same-environment regressions remain blocking. CodSpeed v5 applies this pattern to cross-version comparisons by marking them N/A instead of showing a misleading diff, as described in the v5 announcement.
Activity
vyhuholl commentedon Oct 12, 2026
We see the same in wemake-services/django-modern-rest, with a Python benchmark in Simulation mode on
ubuntu-latest: runner 5.4.0 (CodSpeedHQ/action 5.2.1, then 5.4.0), pytest-codspeed 5.0.3, Python 3.13.16, msgspec 0.22.0.test_msgspec_with_rendererserializes 100 msgspec structs to JSON 10 times. Its input is fixed,pytest-randomlyis off andPYTHONHASHSEED=0. Since 2026-10-06 it has had exactly two results, 5.1 ms and 5.7 ms. Eight PR reports flagged it in that time, each one between those two values and each with "Different runtime environments detected":BASEHEAD.github/lychee.toml, a Markdown fileNone of these changes touch what the benchmark runs. In #1725, both runs used the same image (
ubuntu-24.0420261004.327.1), kernel (6.17.0-1022-azure) and Python, the report: https://app.codspeed.io/wemake-services/django-modern-rest/branches/fix-lycheeMarking these comparisons incompatible, as this issue asks, would have saved all eight.
One thing I couldn't square with the glibc article: on any AVX2 host, valgrind-codspeed answers CPUID from a fixed "Core-i7-4910-like" table, cache leaf 4 and XSAVE leaf 0xD included:
So glibc inside Valgrind should detect the same cache sizes and features on both CPU types. Valgrind also drops auxv entries it doesn't know,
AT_MINSIGSTKSZamong them (initimg-linux.c), the cache simulation is fixed by--I1/--D1/--LL(measure.rs), and ASLR is off (measure.rs). I couldn't find what carries the host CPU into the result. If you know, could you say whetherGLIBC_TUNABLEScan work around it at all?