You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Disk read load stalls the subc frame loop for 30 s; the watchdog exits and every client loses its calls (two incidents on v0.62.0) #891
On a Linux host with heavy disk read load, the subc module stops answering for 30 s, the stall watchdog makes it exit with code 1, and the daemon restarts it. Every connected client loses its in-flight tool calls, and running bash tasks come back as "daemon restarted, process fate unknown". This happened twice in 30 hours on v0.62.0.
04d31f7dc on main ("keep task state reads off the subc frame loop") matches the first incident. It is not in any release yet. Please consider a patch release with it.
Incidents
Both: v0.62.0, subc mode, Linux, btrfs on NVMe.
1. 2026-10-24 00:18:45Z, during a nightly backup upload (IO pressure "full" avg60 rose from 0.7 to 13.7).
Log, in order: bash task retention lock budget exceeded: worst_lock_us=175915 budget_us=100000 (00:17:31), then tool_call unanswered past client deadline class name=bash_drain_completions ... elapsed_ms=25977 (twice) and name=read ... elapsed_ms=25750 deadline_class_ms=2500.
Snapshot: phase=turn_work frame=none scheduler_holder=none; one module thread in D state with wchan folio_wait_bit_common.
2. 2026-10-25 ~06:31Z, during two concurrent cargo test builds (IO pressure "full" avg60 19.4).
Watchdog: stalled_for_ms=30127 phase=turn_work frame=none; exit 1 at 06:31:31Z, registered again at 06:31:36Z.
Snapshot: two threads in D state, one module thread with wchan read_extent_buffer_pages (btrfs metadata read) and one aft-callgraph-r. The last log lines before the stall were tier-2 index builds (category=duplicates elapsed_ms=30493).
What is and is not shown
Incident 1 fits the commit: the calls that stopped answering were bash_drain_completions and read, and a task-state lock was slow just before.
Incident 2 does not name the blocked call. The snapshot has only thread state and wchan, no stack or in-flight tool name, and no "unanswered" lines were logged. So I cannot say if 04d31f7dc covers it, or if another read on the frame loop (for example index or callgraph work) waits on the disk.
Asks
A patch release that includes 04d31f7dc.
Make the stall snapshot name the frame loop's in-flight work (tool or command name, and the phase entry point), so the next exit can be attributed without guessing.
Three more stalls on v0.62.0 on 2026-10-25, all during heavy concurrent build load. Each passed the 15 s snapshot point and recovered before the 30 s exit, so the module did not restart:
The 07:15 case is a different shape from the two incidents above. The snapshot names a holder of the scheduler state lock: in v0.62.0, executor/mod.rs:1829 is the self.inner.state.lock() in the job submit path, held for 15.9 s. No commit since v0.62.0 changes executor/mod.rs on main (checked at 8072df9), so I do not think 04d31f7dc covers this one.
Totals on this host in about 40 hours: two watchdog exits and three stalls of 15 to 30 s. The scheduler_holder field already made the 07:15 case attributable. The turn_work snapshots still do not name the in-flight tool call, which is the second ask above.
Activity
mason: attribute frame-loop stalls and defer watch diagnostics (corte…
mason: make frame-loop stall attribution cheap (cortexkit#891)
iceteaSA commentedon Oct 25, 2026
Three more stalls on v0.62.0 on 2026-10-25, all during heavy concurrent build load. Each passed the 15 s snapshot point and recovered before the 30 s exit, so the module did not restart:
phase=select_wait frame=none scheduler_holder=crates/aft/src/executor/mod.rs:1829 held_for_ms=15910phase=turn_work frame=none scheduler_holder=nonephase=turn_work frame=none scheduler_holder=noneThe 07:15 case is a different shape from the two incidents above. The snapshot names a holder of the scheduler state lock: in v0.62.0,
executor/mod.rs:1829is theself.inner.state.lock()in the job submit path, held for 15.9 s. No commit since v0.62.0 changesexecutor/mod.rson main (checked at 8072df9), so I do not think04d31f7dccovers this one.Totals on this host in about 40 hours: two watchdog exits and three stalls of 15 to 30 s. The
scheduler_holderfield already made the 07:15 case attributable. Theturn_worksnapshots still do not name the in-flight tool call, which is the second ask above.