Repository navigation
fix(railway): the watchdog still cannot go red — classify can never fail the job, and the read path has no retry #6474
Description
Follow-ups from the #6469 review. #6469 fixed the cause that was firing; these are the mechanism that will hide the next one, plus three inaccurate claims that shipped with it.
Found by six review lenses plus an independent cross-model pass. The two P1s are pre-existing code that #6469 made reachable at volume for the first time.
P1 — classify structurally cannot fail the job
scripts/dispatch-stale-railway-reconcile.mjs:1707
if (phase !== 'classify') process.exitCode = 1;classify is unconditionally excluded from ever setting a non-zero exit code, even when main() throws, and no step in railway-deploy-trigger-watchdog.yml gates on steps.controller.outputs.outcome. Budget exhaustion, the 10-page cap, or one un-retried 403 all produce warning -> exit 0 -> green check.
This is the same green-while-dead shape that hid the pagination bug for months. #6469 made it fire less often; it did not close it.
Fix: fail the classify job (or emit a distinct annotation a monitor consumes) when N consecutive scheduled runs return DEFERRED_AMBIGUOUS with a GITHUB_READ_* reason. The watchdog already reads its own workflow history via readSourceRunIdentity and owns a durable control plane, so it can detect the streak itself. Update the process-contract test at tests/railway-stale-reconcile-dispatch.test.mjs:1866, which currently asserts perpetual exit 0 as a contract.
P1 — read path has no retry or backoff for transient failures
scripts/dispatch-stale-railway-reconcile.mjs:269 — every non-ok GET resolves to GITHUB_READ_FAILED with no retry branch, while the POST/dispatch path already implements the definitive-vs-ambiguous split and correctly treats 403 secondary-rate-limit as transient.
One transient 403 mid-walk aborts the whole read and lands in the silent tier above. Budget context: 4 runs/hour x WATCHDOG_MAX_API_REQUESTS=250 sits at the ~1000/hr per-repo GITHUB_TOKEN ceiling with no headroom for other workflows. Measured today: 91 requests for a full snapshot with 62/233 non-success runs, so a ~3.75x failure storm — exactly the condition the watchdog exists for — exhausts the budget and silences it.
Fix: bounded retry with exponential backoff and jitter for retryable GET failures (403 secondary-rate-limit, 429, 5xx), honoring Retry-After / x-ratelimit-reset.
P1 — readNewestGate can silently truncate and report the gate as missing
readNewestGate (:417) is the only #paginate caller that passes no readTotalCount, so the frozen-total_count, duplicate-id, and truncation invariants are all inert — on the one collection that actually paginates in production. A short or Link-less origin response returns a partial prefix and the walk reports state: 'missing'.
Reproduced locally during review. Live margin is thin: the newest gate sits at index 79 of page 1 on current main. It is fail-closed today ('missing' -> DEFERRED_NON_GREEN_MAIN), but wrong-and-green.
Fix (also removes a page of I/O per snapshot): statuses are newest-first, so stop the walk at the first context === 'gate' match via a stopWhen predicate. Add a fail-closed rule for the no-gate case: if the walk ends with no gate and the final page was exactly GITHUB_PAGE_SIZE items, throw GITHUB_READ_FAILED rather than returning 'missing'. Sibling precedent: scripts/resolve-railway-reconcile-control.mjs:375 terminates on batch.length < 100 — a body-derived condition an origin cannot spoof.
P2 — the mock shape that hid the bug is still the suite default
tests/railway-stale-reconcile-dispatch.test.mjs:702, :718, :726 still build Link headers from the request pathname:
link: `<https://api.github.com${parsed.pathname}?per_page=100&page=2>; rel="next"`Confirmed during review to pass against the pre-fix buggy code. #6469 added the production /repositories/<id>/... form for the statuses collection only; runs, jobs, the fail-closed table and the budget test remain blind.
Fix: build the link inside the shared json() helper (:656) as /repositories/<numeric-id>/<rewritten-path> by default, so the whole suite runs against the production shape instead of one statuses-only test a future editor can normalize away.
P3 — three inaccurate claims shipped in #6469
scripts/dispatch-stale-railway-reconcile.mjs:378-379— the comment asserts "Completeness stays enforced below by the frozen total_count, duplicate-id, and truncation invariants." All three are skipped forreadNewestGate. Name the controls that actually apply:WATCHDOG_MAX_PAGESand the fail-closed'missing'mapping.scripts/dispatch-stale-railway-reconcile.mjs:377and the fix(railway): watchdog never paginates — GitHub rewrites next links to the repository-ID path (#6378) #6469 PR body — both frame the change as removing an origin-controlled routing input. A security pass probed 13 off-allowlist payloads:#authorizeblocked every one on its own, and the deleted check pinned pathname and page arithmetic on top of that. There was no routing exposure; this is a correctness fix and should be described as one.docs/railway-seed-consolidation-runbook.md:147-148— the table still says "the deploy-scopedRAILWAY_RECONCILE_DEPLOY_TOKEN_V2" and "read-only Railway Viewer token", seven lines above the paragraph added in the same PR establishing that no Railway token carries a scope and that the Viewer token is read-only by convention only.
P3 — unreachable guard
:386 — !Number.isSafeInteger(currentPage) || currentPage < 1 cannot fire: all four callers start at literal page=1, #authorize gates page format, and WATCHDOG_MAX_PAGES caps the increment. An orphaned fragment of the removed compound comparison, carrying a "not exact" message from a check that no longer exists. Drop it, or document it as forward-looking and give it a fixture that forges an out-of-range value.
Activity
docs(railway): correct three claims that shipped with the pagination …
fix(railway): let the watchdog go red, retry the read path, and stop …
fix(railway): let the watchdog go red, retry the read path, and stop …
fix(railway): let the watchdog go red, retry the read path, and stop …
fix(railway): let the watchdog go red, retry the read path, and stop …