Skip to content

fix(test): make a bare vitest invocation use the isolated CI config - #660

Merged
Ark0N merged 4 commits into
Ark0N:masterfrom
opticon454:fix/test-home-isolation
Oct 22, 2026
Merged

Ark0N merged 4 commits into
Ark0N:masterfrom
opticon454:fix/test-home-isolation

Conversation

@opticon454

Copy link
Copy Markdown
Contributor

What

A bare npx vitest run writes test fixtures into the developer's real ~/.codeman.

Why

There is no root vitest config: the configs live in config/ and npm test passes --config. A bare invocation therefore has no setupFiles, test/setup.ts never runs, HOME is not redirected and the suite works against the real data dir. On a machine that runs Codeman this left a clis.json containing a test entry, a fixture users.json, webhook.json, intents.json, .env, settings.json values and a .secret-fixture file; the running instance loaded them on its next restart, which looks exactly like "my settings were reset".

How

  • A root vitest.config.ts re-exports config/vitest.ci.config.ts, so the default invocation is the isolated one. npm test and the browser, mobile and perf scripts keep passing their own --config.
  • The shared temp-HOME teardown keeps removing the folder (up to ~5 s) until a straggling writer goes quiet, and leaves it to the exit backstop instead of failing a file that passed. On a machine with a real agent CLI installed, a test's code path probes or spawns claude, which flushes its own ~/.claude.json and ~/.claude/backups/ into the temp HOME after the test ends; rmSync then raced it and failed the whole file with ENOTEMPTY after every test had passed (about one run in three for session-routes-claude-model on unmodified master; CI has no such CLI). Twelve consecutive runs of that file now pass.
  • Docs: CLAUDE.md and Contributing state that a bare npx vitest run is isolated now, and what it used to do.

Tests

test/test-env-isolation.test.ts pins that the root config exists and loads test/setup.ts. Checked by running a test that used to write into the data dir with a bare npx vitest run and confirming nothing appeared there. Changeset included.

Fixes #640

opticon454 and others added 4 commits October 17, 2026 05:07
`npx vitest run` with no --config found no config file, so test/setup.ts never ran and HOME
was not redirected: the suite wrote fixtures (clis.json with a test entry, users.json,
webhook.json, intents.json, .env, settings.json values) into the developer's real ~/.codeman,
and a running Codeman then loaded them on its next restart.

A root vitest.config.ts re-exports config/vitest.ci.config.ts, so the default invocation is the
isolated one; npm test and the browser/mobile/perf scripts keep passing their own --config. The
shared temp-HOME teardown also retries on ENOTEMPTY, which failed whole files intermittently
after every test had passed.
CLAUDE.md and Contributing said raw npx vitest skips the config; state what happened when it did
(fixtures written into the real ~/.codeman) and that the root vitest.config.ts now makes a bare run
isolated, with the test that pins it.
maxRetries alone was not enough: on a machine with a real agent CLI installed, a test's code path
probes or spawns `claude`, which flushes its own ~/.claude.json and ~/.claude/backups/ into the
temp HOME a moment after the test ends, so rmSync raced it and failed the whole file with ENOTEMPTY
after every test had passed (about one run in three for session-routes-claude-model; CI has no CLI).

Retry the removal for up to ~5s, and if it never goes quiet leave the folder to the exit backstop
instead of failing a file that passed. 12 consecutive runs of that file: no failure.
safeRmHomeTree and isUnderTestHome answered "is this path under process.env.HOME?".
With test/setup.ts skipped (a bare vitest) HOME is the developer's real home, so every
real file passed, which is how the real linked-cases registry was deleted on 2026-10-10.

They now also require HOME to be the throwaway codeman-vitest-* directory setup.ts
creates, and the shared route harness throws when it is not, so every route test refuses
to start against a real ~/.codeman rather than just the one file that checked.
@Ark0N

Ark0N commented Oct 20, 2026

Copy link
Copy Markdown
Owner

Thanks for this, @opticon454. It closes a nasty trap: a bare npx vitest run used to skip test/setup.ts and write fixtures into the real ~/.codeman. The root vitest.config.ts re-export fixes that, and the stricter deletion guards close the hole that let a run without setup delete real state. I checked vitest's config lookup (it walks up from cwd), ran a bare npx vitest run from the repo root and from test/ (both land in the sandboxed HOME), and confirmed packages/xterm-zerolag-input still picks up its own config. CI is green.

A few small things I will apply at merge time, so no action is needed unless you want to do them yourself:

  1. test/setup.ts:130: the retry loop has no wall-clock bound, and afterAll runs under vitest's default 10 s hookTimeout. Each failed attempt is the 250 ms wait plus rmSync's own retry sleeps (50 + 100 + 150 ms), so 20 attempts can run past 10 s. A temporary test that left a writer filling $HOME/.claude/backups for 20 s failed with Hook timed out in 10000ms after its test passed, so the "leave it to the exit backstop" branch is never reached. I will bound the loop with a deadline (around 5 s) and keep the comment's claim in line with it.
  2. CLAUDE.md:338 and docs/architecture-invariants.md:1175 still say a raw npx vitest skips setup.ts, which now contradicts your updated note at CLAUDE.md:334. I will reword both.
  3. docs/architecture-invariants.md:1173 still describes safeRmHomeTree as refusing only paths outside HOME. I will add the sandbox-HOME requirement, the route-harness assert and the 2026-10-10 incident there.
  4. CLAUDE.md:9: the preamble lists every file allowed at the repo root and why. I will add vitest.config.ts (vitest resolves its config by walking up from cwd) so a future tidy-up does not move it into config/ and reopen Bare npx vitest run writes test fixtures into the real ~/.codeman #640.

Smaller wording items: the assertSandboxedHome JSDoc (test/mocks/test-helpers.ts:64) says every route test refuses to start, but only the 39 route files on createRouteTestHarness do (the 30 that build their own app from createMockRouteContext are not covered), and the changeset does not mention the stricter guards. I will adjust both.

Thanks again for tracking down both the root cause and the ENOTEMPTY flake.

@Ark0N
Ark0N merged commit 710ed73 into Ark0N:master Oct 22, 2026
2 checks passed
Ark0N pushed a commit that referenced this pull request Oct 22, 2026
…count

Merge-time fix for #660. afterAll runs under vitest's 10 s hookTimeout, and
each failed removal attempt costs rmSync's own retry sleeps (50 + 100 + 150
ms) plus the 250 ms wait, so 20 attempts overran it: with a writer still
filling $HOME/.claude/backups, the file failed with "Hook timed out in
10000ms" after its test passed, and the "leave it to the exit backstop" branch
was never reached. The loop now stops after about 5 s. Checked with a
temporary (uncommitted) test that leaves a child process writing there for
20 s: before, the file failed at 11 s with the hook timeout; after, it passes
in under 6 s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Ark0N pushed a commit that referenced this pull request Oct 22, 2026
…ndbox HOME

Merge-time fix for #660. CLAUDE.md and architecture-invariants still said a
raw npx vitest skips setup.ts, contradicting the updated Testing note: both
now say it resolves the root vitest.config.ts (and why that file must stay).
The architecture-invariants entry for safeRmHomeTree adds the sandbox-HOME
requirement (isSandboxedHome, TEST_HOME_PREFIX), the same rule for
isUnderTestHome, the route-harness assert and the 2026-10-10 incident.
CLAUDE.md's repo-root preamble lists vitest.config.ts with its reason, so a
tidy-up does not move it into config/ and reopen #640. The
assertSandboxedHome JSDoc no longer claims every route test is covered: only
the ones built on createRouteTestHarness are.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Ark0N pushed a commit that referenced this pull request Oct 22, 2026
… the isolated CI config

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Ark0N

Ark0N commented Oct 22, 2026

Copy link
Copy Markdown
Owner

Merged and shipped in 1.44.0. Thank you @opticon454! This closes a trap that had already cost real state once. The release gate even proved the point: the one file that failed on a branch without your change failed on exactly the ENOTEMPTY teardown flake this PR fixes.

At merge I bounded the teardown retry by a deadline of about 5 s instead of an attempt count, so a stuck writer can no longer push afterAll past vitest's 10 s hook timeout (a temporary test with a writer running for 20 s failed with a hook timeout before and passes in under 6 s now). I also brought CLAUDE.md and the invariants in line with your note: a bare npx vitest is isolated now, vitest.config.ts is listed among the files that must stay at the repo root, and the deletion guards and the 2026-10-10 incident are described. The assertSandboxedHome JSDoc now says which route tests it covers. Closes #640.

@opticon454
opticon454 deleted the fix/test-home-isolation branch October 22, 2026 20:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bare npx vitest run writes test fixtures into the real ~/.codeman

3 participants