vitest-plugin: root-level beforeAll/afterAll run twice in analysis mode #106

Summary

In analysis (simulation) mode, a benchmark file's top-level beforeAll and afterAll hooks run twice. Hooks inside a describe run once, as expected.

Cause

Vitest's runner (@vitest/runner, runSuite) calls the file suite's beforeAll itself, then hands the suite to a custom runner's runSuite, then calls afterAll. This is the same on Vitest 3.2 and 4.1:

beforeAllCleanups = await callSuiteHook(suite, suite, "beforeAll", runner, [suite]);
if (runner.runSuite) {
  await runner.runSuite(suite);
} else { /* … */ }
// finally: afterAll

AnalysisRunner.runSuite then calls runAnalysisBenchmarkSuite(rootSuite), which calls callSuiteHook(suite, suite, "beforeAll") / "afterAll" on that same root suite again (packages/vitest-plugin/src/legacy/analysis.ts). The setup therefore runs once outside the CodSpeed run and once inside it.

Impact

Any benchmark whose top-level setup has side effects does that work twice: seeding a database, starting a server, opening a port, or writing fixtures to disk. In our case the second beforeAll reassigned a module-level fixture path. Objects created by the first call were then driven with paths from the second, and a benchmark waiting on one of them hung until CI killed the job.

Reproduction

import { beforeAll, bench } from "vitest";

let calls = 0;
beforeAll(() => {
  calls += 1;
  console.log("beforeAll", calls);
});

bench("noop", () => {});

CODSPEED_ENV=1 CODSPEED_RUNNER_MODE=simulation vitest bench --run prints beforeAll 1 and beforeAll 2. Without the plugin, it prints only beforeAll 1.

Reproduced with @codspeed/vitest-plugin@5.7.1 on Vitest 4.1.11. 6.1.0's dist/legacy/analysis.mjs contains the same root-level callSuiteHook(suite, suite, "beforeAll").

Suggested fix

Skip the hook calls for the root suite in runAnalysisBenchmarkSuite, since Vitest already ran them. Keep them for nested suites, which are only reached through this runner. I'll open a PR with a regression test.