Skip to content

Disk read load stalls the subc frame loop for 30 s; the watchdog exits and every client loses its calls (two incidents on v0.62.0) #891

Open

Description

@iceteaSA

Summary

On a Linux host with heavy disk read load, the subc module stops answering for 30 s, the stall watchdog makes it exit with code 1, and the daemon restarts it. Every connected client loses its in-flight tool calls, and running bash tasks come back as "daemon restarted, process fate unknown". This happened twice in 30 hours on v0.62.0.

04d31f7dc on main ("keep task state reads off the subc frame loop") matches the first incident. It is not in any release yet. Please consider a patch release with it.

Incidents

Both: v0.62.0, subc mode, Linux, btrfs on NVMe.

1. 2026-10-24 00:18:45Z, during a nightly backup upload (IO pressure "full" avg60 rose from 0.7 to 13.7).

  • Log, in order: bash task retention lock budget exceeded: worst_lock_us=175915 budget_us=100000 (00:17:31), then tool_call unanswered past client deadline class name=bash_drain_completions ... elapsed_ms=25977 (twice) and name=read ... elapsed_ms=25750 deadline_class_ms=2500.
  • Watchdog: stall watchdog: exiting marker=subc_frame_loop stalled_for_ms=30302 phase=turn_work.
  • Snapshot: phase=turn_work frame=none scheduler_holder=none; one module thread in D state with wchan folio_wait_bit_common.

2. 2026-10-25 ~06:31Z, during two concurrent cargo test builds (IO pressure "full" avg60 19.4).

  • Watchdog: stalled_for_ms=30127 phase=turn_work frame=none; exit 1 at 06:31:31Z, registered again at 06:31:36Z.
  • Snapshot: two threads in D state, one module thread with wchan read_extent_buffer_pages (btrfs metadata read) and one aft-callgraph-r. The last log lines before the stall were tier-2 index builds (category=duplicates elapsed_ms=30493).

What is and is not shown

  • Incident 1 fits the commit: the calls that stopped answering were bash_drain_completions and read, and a task-state lock was slow just before.
  • Incident 2 does not name the blocked call. The snapshot has only thread state and wchan, no stack or in-flight tool name, and no "unanswered" lines were logged. So I cannot say if 04d31f7dc covers it, or if another read on the frame loop (for example index or callgraph work) waits on the disk.

Asks

  1. A patch release that includes 04d31f7dc.
  2. Make the stall snapshot name the frame loop's in-flight work (tool or command name, and the phase entry point), so the next exit can be attributed without guessing.

Activity

  1. iceteaSA commented on Oct 25, 2026

    @iceteaSA
    ContributorAuthor

    Three more stalls on v0.62.0 on 2026-10-25, all during heavy concurrent build load. Each passed the 15 s snapshot point and recovered before the 30 s exit, so the module did not restart:

    Time (UTC) Snapshot header Threads in D state
    07:15:34 phase=select_wait frame=none scheduler_holder=crates/aft/src/executor/mod.rs:1829 held_for_ms=15910 none listed
    08:46:03 phase=turn_work frame=none scheduler_holder=none one bash-task worker, two inspect workers
    17:33:23 phase=turn_work frame=none scheduler_holder=none one executor worker, one inspect worker

    The 07:15 case is a different shape from the two incidents above. The snapshot names a holder of the scheduler state lock: in v0.62.0, executor/mod.rs:1829 is the self.inner.state.lock() in the job submit path, held for 15.9 s. No commit since v0.62.0 changes executor/mod.rs on main (checked at 8072df9), so I do not think 04d31f7dc covers this one.

    Totals on this host in about 40 hours: two watchdog exits and three stalls of 15 to 30 s. The scheduler_holder field already made the 07:15 case attributable. The turn_work snapshots still do not name the in-flight tool call, which is the second ask above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions