Skip to content

fix(railway): the watchdog still cannot go red — classify can never fail the job, and the read path has no retry #6474

Closed

Description

@koala73

Follow-ups from the #6469 review. #6469 fixed the cause that was firing; these are the mechanism that will hide the next one, plus three inaccurate claims that shipped with it.

Found by six review lenses plus an independent cross-model pass. The two P1s are pre-existing code that #6469 made reachable at volume for the first time.

P1 — classify structurally cannot fail the job

scripts/dispatch-stale-railway-reconcile.mjs:1707

if (phase !== 'classify') process.exitCode = 1;

classify is unconditionally excluded from ever setting a non-zero exit code, even when main() throws, and no step in railway-deploy-trigger-watchdog.yml gates on steps.controller.outputs.outcome. Budget exhaustion, the 10-page cap, or one un-retried 403 all produce warning -> exit 0 -> green check.

This is the same green-while-dead shape that hid the pagination bug for months. #6469 made it fire less often; it did not close it.

Fix: fail the classify job (or emit a distinct annotation a monitor consumes) when N consecutive scheduled runs return DEFERRED_AMBIGUOUS with a GITHUB_READ_* reason. The watchdog already reads its own workflow history via readSourceRunIdentity and owns a durable control plane, so it can detect the streak itself. Update the process-contract test at tests/railway-stale-reconcile-dispatch.test.mjs:1866, which currently asserts perpetual exit 0 as a contract.

P1 — read path has no retry or backoff for transient failures

scripts/dispatch-stale-railway-reconcile.mjs:269 — every non-ok GET resolves to GITHUB_READ_FAILED with no retry branch, while the POST/dispatch path already implements the definitive-vs-ambiguous split and correctly treats 403 secondary-rate-limit as transient.

One transient 403 mid-walk aborts the whole read and lands in the silent tier above. Budget context: 4 runs/hour x WATCHDOG_MAX_API_REQUESTS=250 sits at the ~1000/hr per-repo GITHUB_TOKEN ceiling with no headroom for other workflows. Measured today: 91 requests for a full snapshot with 62/233 non-success runs, so a ~3.75x failure storm — exactly the condition the watchdog exists for — exhausts the budget and silences it.

Fix: bounded retry with exponential backoff and jitter for retryable GET failures (403 secondary-rate-limit, 429, 5xx), honoring Retry-After / x-ratelimit-reset.

P1 — readNewestGate can silently truncate and report the gate as missing

readNewestGate (:417) is the only #paginate caller that passes no readTotalCount, so the frozen-total_count, duplicate-id, and truncation invariants are all inert — on the one collection that actually paginates in production. A short or Link-less origin response returns a partial prefix and the walk reports state: 'missing'.

Reproduced locally during review. Live margin is thin: the newest gate sits at index 79 of page 1 on current main. It is fail-closed today ('missing' -> DEFERRED_NON_GREEN_MAIN), but wrong-and-green.

Fix (also removes a page of I/O per snapshot): statuses are newest-first, so stop the walk at the first context === 'gate' match via a stopWhen predicate. Add a fail-closed rule for the no-gate case: if the walk ends with no gate and the final page was exactly GITHUB_PAGE_SIZE items, throw GITHUB_READ_FAILED rather than returning 'missing'. Sibling precedent: scripts/resolve-railway-reconcile-control.mjs:375 terminates on batch.length < 100 — a body-derived condition an origin cannot spoof.

P2 — the mock shape that hid the bug is still the suite default

tests/railway-stale-reconcile-dispatch.test.mjs:702, :718, :726 still build Link headers from the request pathname:

link: `<https://api.github.com${parsed.pathname}?per_page=100&page=2>; rel="next"`

Confirmed during review to pass against the pre-fix buggy code. #6469 added the production /repositories/<id>/... form for the statuses collection only; runs, jobs, the fail-closed table and the budget test remain blind.

Fix: build the link inside the shared json() helper (:656) as /repositories/<numeric-id>/<rewritten-path> by default, so the whole suite runs against the production shape instead of one statuses-only test a future editor can normalize away.

P3 — three inaccurate claims shipped in #6469

  1. scripts/dispatch-stale-railway-reconcile.mjs:378-379 — the comment asserts "Completeness stays enforced below by the frozen total_count, duplicate-id, and truncation invariants." All three are skipped for readNewestGate. Name the controls that actually apply: WATCHDOG_MAX_PAGES and the fail-closed 'missing' mapping.
  2. scripts/dispatch-stale-railway-reconcile.mjs:377 and the fix(railway): watchdog never paginates — GitHub rewrites next links to the repository-ID path (#6378) #6469 PR body — both frame the change as removing an origin-controlled routing input. A security pass probed 13 off-allowlist payloads: #authorize blocked every one on its own, and the deleted check pinned pathname and page arithmetic on top of that. There was no routing exposure; this is a correctness fix and should be described as one.
  3. docs/railway-seed-consolidation-runbook.md:147-148 — the table still says "the deploy-scoped RAILWAY_RECONCILE_DEPLOY_TOKEN_V2" and "read-only Railway Viewer token", seven lines above the paragraph added in the same PR establishing that no Railway token carries a scope and that the Viewer token is read-only by convention only.

P3 — unreachable guard

:386 — !Number.isSafeInteger(currentPage) || currentPage < 1 cannot fire: all four callers start at literal page=1, #authorize gates page format, and WATCHDOG_MAX_PAGES caps the increment. An orphaned fragment of the removed compound comparison, carrying a "not exact" message from a check that no longer exists. Drop it, or document it as forward-looking and give it a fixture that forges an out-of-range value.

Refs #6469, #6378.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions