Repository navigation
fix(cases): bound path probes for linked workspaces and session creation, so an unreachable mount cannot freeze the server - #516
Conversation
|
DRAFT_COMMENT |
|
Hi @Ark0N, it looks like your comment here came through as the placeholder text "DRAFT_COMMENT", so I think the actual review didn't post. Happy to pick it up whenever you get a chance to re-send it. |
|
Thanks for this, @aakhter, and sorry for the "DRAFT_COMMENT" placeholder that went out here on 10-01. That was a glitch in my review tooling, and this is the review that should have been posted. This PR stops a linked case on an unreachable network mount from freezing the server. What needs another round is how callers read a non-answer. Today "did not answer in 1.5 s" and "refused by the cap" both come back as Must fix
Should fix
Small ones
Once items 1 and 2 are in with their tests, this gets another review, and 3 and 4 are one-line swaps that can ride the same push. Thanks again for chasing this one down. The freeze is real, and the hard part of the fix is already done well. |
…t cannot freeze the server A linked case can live on a network mount. When that mount goes away, a hard mount makes stat() wait indefinitely, and the existsSync() probes in the case routes and the workspace hook/statusline helpers ran on the event loop, so a single GET /api/cases (or a session create in that workspace) froze the whole web server until the mount came back. Add boundedPathExists() (src/utils/bounded-path-probe.ts): an async stat that answers "absent" after 1.5 s, shares one in-flight probe per path, remembers a timed-out path until its stat finally settles, and refuses to start new probes while two stalled ones still hold libuv threadpool workers. Route the read-side probes in case-routes.ts and hooks-config.ts through it. The settings writers in hooks-config.ts use an async lstat that treats only ENOENT as missing, so an unreachable workspace is never mistaken for an empty one and has its settings recreated.
…all cap
The bounded path probe answered "absent" both when a path did not exist and
when it simply did not answer, so a stalled linked case 404'd and the Run
button scaffolded a stray local case over it, and two stalled paths anywhere
made every unrelated path read as absent (hooks skipped, statusLine
overridden, the clone warning lost).
- probePath()/probePathKind() are tri-state: present (or directory/file),
absent (ENOENT/ENOTDIR only) and unknown (timeout, other errors, refusal).
boundedPathExists() stays as the display-only boolean.
- A stalled path takes only its own mount out of probing (deepest mount
point from /proc/self/mounts, never /; just the path itself when there is
no mount table). Unrelated paths keep probing. The process-wide cap is a
backstop that answers unknown, and a single-path user request can probe
past it ({ pastCap: true }), still bounded and still recorded as stalled.
One console.warn when a path first stalls and one when the cap engages.
- GET /api/cases/:name keeps NOT_FOUND for definite absence only. An
unreachable linked case answers with its registered path and
unreachable: true; a local one answers OPERATION_FAILED. runClaude and
runShell create a case only on errorCode NOT_FOUND. The case list keeps an
unreachable linked case, marked unreachable, instead of dropping it, and
fix-plan reports an unreadable plan as an error, not "no plan".
- applyWorkspaceHooks and the statusLine helpers skip only a workspace that
is absent or on the stalled mount; a capacity refusal no longer stops
hooks being installed elsewhere, and an unreadable settings file never
lets the exporter override a user's own statusLine.
- The clone flow's repo-settings warning is back on its synchronous check,
and stripCaseEnvKeys uses pathExistsForWrite.
- POST /api/sessions (workingDir) and POST /api/quick-start (case folder)
probe with the bounded probe instead of statSync/existsSync. Missing and
non-directory keep INVALID_INPUT; unknown is OPERATION_FAILED, and
quick-start never scaffolds over a folder that did not answer.
- PATH_PROBE_TIMEOUT_MS and MAX_STALLED_PATH_PROBES move to
src/config/path-probe.ts, overridable via CODEMAN_PATH_PROBE_TIMEOUT_MS
(default 1500) and CODEMAN_PATH_PROBE_MAX_STALLED (default 3), and are
documented in the Settings Reference.
- The probe is exported from the utils barrel and imported from there.
cf9748e to
d1bfbb4
Compare
|
Thanks for the careful review, and no worries at all about the DRAFT_COMMENT placeholder; this was well worth the wait. The throwaway tests on my own route harness made both must-fix items very concrete. I've rebased onto 1.34.0 and pushed one new commit on top of the original (d1bfbb4):
The title and description now cover session creation too. Every new test failed before its fix, and reverting each of the fixes for items 1, 2, 3 and 5 makes its tests fail. Typecheck, lint, format, the frontend-syntax and browser-exclude checks, and the test files touching these modules all pass. |
|
Thanks for the quick and careful turnaround, @aakhter. This PR makes every probe of a linked case folder or session workingDir bounded and tri-state, so an unreachable network mount can no longer freeze the server. All seven items from the last round are in, each with a test, and typecheck, lint, format, the frontend checks and the full test suite pass here. Three edge cases in the new mount-scoping and 1. Under the stall cap, 2.
Please give 3. The stall scope cannot see through symlinks ( Two small ones that can ride along:
Once 1 to 3 are in with their tests, this is ready to merge. Thanks again: the freeze is real, and the probe itself (timers, in-flight sharing, no stray rejections) is in good shape. |
…s to network mounts - applyWorkspaceHooks: an "unknown" probe that is not near a stalled path (refused by the stall cap, or an unexpected stat error) no longer reads as "go ahead". It checks existence with pathExistsForWrite first, so a deleted workspace is not recreated by the mkdir -p in ensureCodemanHooks. - pastCap gets a hard ceiling, PATH_PROBE_STALL_CEILING = UV_THREADPOOL_SIZE (default 4) minus one, so explicit requests against several dead paths can never take the last libuv worker. The bulk cap now defaults to one below the ceiling (2 with the default pool), leaving a slot for an explicit request. - A stall widens to its mount only for network and FUSE filesystem types read from /proc/self/mounts; on a local mount (a path typed under a local /home that reaches a NAS through a symlink) it narrows to the stalled path. - GET /api/cases/:name probes CLAUDE.md with pastCap, like the folder probe. - Comment in config/path-probe.ts describes the mount-scoped stall.
|
Thanks for another careful round, and for reproducing each case. All five are in 9fa4410.
Each new test failed before its fix, and reverting each fix fails it again. Typecheck, lint, format, the frontend checks and the test files touching these modules all pass. Since round 1 the description has said the default cap is 3; with this round the bulk default is back to 2 and the explicit-request ceiling is 3. I've updated the description to match. |
- hooks-config: a probe the bulk cap refused gets ONE bounded re-probe past the cap (probeBeforeTouching), and whatever is still unknown is skipped. The per-spawn hook and statusLine helpers used to fall back to an unbounded lstat/readFile there, which on a dead workspace never settled and could take the last threadpool workers (and hang the boot hook sweep). New test: cap engaged, stat/lstat/readFile hanging on two more paths; both helpers return. - describeUnknownPath()/unknownPathReason(): POST /api/sessions, quick-start and GET /api/cases/:name now say a folder was not checked (other mounts are still not answering) instead of blaming a healthy folder at the stall ceiling. errorCodes unchanged. - #535 x #516: Create in a custom folder probes the parent through the bounded probe before realpath/stat/lstat/readdir touch it; an unknown parent is 422 OPERATION_FAILED (UNREACHABLE) within the probe timeout. New test. - Docs: MAX_STALLED default is 2 (follows UV_THREADPOOL_SIZE), CaseInfo .unreachable covers a refused probe, the boot sweep skips an unanswering workspace, a CLAUDE.md gotcha for bounded probes, verbs.md documents the 422 (plugin mirror synced), api-reference documents the custom-folder 422. - Tests: the launcher case-lookup describe is no longer nested in the Grok block, and the cap-below-ceiling test no longer depends on an inherited UV_THREADPOOL_SIZE / CODEMAN_PATH_PROBE_MAX_STALLED. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Merged in 1.35.0, thanks @aakhter! A dead mount freezing the whole server is the kind of bug people hit once and never forget, and the tri-state probe with "unknown is never absent" is exactly the right shape. The bot's last review found one remaining hole, which I fixed on the way in (88f5a43):
|
The paste-image route wrote under session.workingDir via ensureUploadDir with no bounded probe, so an upload onto a stalled network mount held one of libuv's few threadpool workers until the mount answered. Probe the workspace with probePathKind past the stall cap before the first disk touch and refuse an unknown mount through describeUnknownPath, the way POST /api/sessions already does.
… review, #516) Clone ran an unbounded realpath over every Docker case workspace, and the mkdir route inherited the picker resolver's realpathSync, which freezes the whole server on a mount that stopped answering. The picker's browse route has the same call and is left as it was. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… it (release review) The round-robin walk capped what it listed, not what it read: every queued folder was read whole, sorted and held until drained, so one GET could pull millions of entries (100 folders of 50,000 files killed a 400 MB heap). A read now keeps only the entries the listing can use, and MAX_WALK_ENTRIES_READ stops opening folders at 250,000 entries, leaving the rest queued and flagged. A folder also spent its whole share on its own files, leaving nothing for the subfolders it had just found, alone in its round or beside a sibling, so search hits inside them were lost. Its files now count as one more share beside those subfolders. Math.max(directoriesInRound, 2) would have fixed only the sole-folder case and cut a wide root's folder list short. Folders below the walk root are probed before their readdir (#516), since the walk reaches folders the depth-first one never did. Filtering at the read also clears the false flag on a folder whose only unread entries were hidden or excluded, and an exactly-full workspace now finishes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s an endpoint A claude session on its own CLAUDE_CONFIG_DIR launched on an endpoint with <isolated>/projects linked to the server's tree. The resume pin looks there, so apply found no transcript and started a new conversation, and on clear the endpoint's transcript stayed behind in the server's tree. The link now goes to <own dir>/projects when the session set one, so the conversation and what the endpoint run adds stay on the session's own account across an apply, a clear and a restart. It goes through linkRealHomeEntry(), which also creates a missing projects folder instead of leaving a dangling link, in the own dir and in ~/.claude/projects on the default path. A relative CLAUDE_CONFIG_DIR of the server gets no link, since joining it would create projects under the server's working directory. Rule #516: guardSessionConfigDirHome() now covers configDirVar, so all three apply sites (route, quick-start, boot recovery) probe the own dir bounded before the apply touches it, and a dir that does not answer falls back to the server's tree. Under test, HOME is a temp dir and CLAUDE_CONFIG_DIR is stripped by test/setup.ts, which a new test pins. Measured on a real Codeman server with a fake claude: apply and re-apply resume under the isolated dir with the transcript landing in the own dir, a clear resumes on the own dir, the isolated dir goes while the transcript stays, and the same holds across a server restart. The documented limits come out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… review) A subfolder whose bounded probe answered `unknown` came back as `children: []` with no `incomplete` and `truncated` false, so a dead mount, or a healthy one slower than the 1.5 s probe on first touch, read as an empty folder and a search missed it in silence. `unknown` is never `absent` (#516). Such a folder now goes on the walk's `unread` list, which flags the node and sets `truncated` in both walks. A folder the filesystem says is gone drops out quietly as before. The row said "file limit reached", which would be false here, so it now reads "Not fully scanned" for either reason. Reproduced on a real disk with a parent at mode 0444, where the child's stat answers EACCES: the API, the search and the panel all report it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A running Codeman remembers a missing CLI for up to 5 minutes, so a status poll never spawns a login shell, and its Run button reads that same memory. So "Run fails, add the agent, Run again" kept saying the CLI was not installed, right after the app had said it was ready. install.sh now touches `cli-installed` in the data dir once a CLI has landed, and resolve() leaves the backoff when that file's mtime changes. It is one stat, inside the backoff branch, and buys one run of the chain, so a resolved CLI never pays for it and a CLI that is still missing goes back to waiting. Re-walking PATH during the backoff would have found hand installs too, but it would put a synchronous stat of every PATH entry on the request path, where one dead network mount freezes the server (Ark0N#516), and execute candidates that were already refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A linked case can live on a network mount. When that mount goes away, a hard mount makes
stat()wait indefinitely. The synchronous probes in the case routes, the workspace hook and statusLine helpers, and session creation (POST /api/sessionsworkingDir,POST /api/quick-startcase folder) then froze the whole server.The probe. The fix is a bounded, tri-state path probe (
src/utils/bounded-path-probe.ts): present, absent (ENOENT/ENOTDIR only) or unknown (no answer within the timeout, another error, or refused).Callers never treat "unknown" as "absent".
GET /api/cases/:namekeeps NOT_FOUND for definite absence. An unreachable linked case comes back with its registered path andunreachable: true, and the Run button creates a case only on NOT_FOUND.Tuning. The timeout and the stall cap live in
src/config/path-probe.ts, overridable withCODEMAN_PATH_PROBE_TIMEOUT_MS(default 1500) andCODEMAN_PATH_PROBE_MAX_STALLED(default 2). An explicit single-path request (opening a case, starting a session) may go up to the libuv threadpool size minus one (3 by default, followingUV_THREADPOOL_SIZE), so one worker always stays free. A slow but healthy mount, such as an sshfs that needs a couple of seconds on first touch, can raise the timeout.Tests
GET /api/cases/:nameon the timeout path, fix-plan, clone, and both session-create routes. They simulate a hard mount as a frozen synchronous call plus a never-settling async stat.