Repository navigation
fix: retry accepted work-queue launch verification safely - #69230
Conversation
Bound transient verification reads by the existing reconciliation policy, retain independently verified evidence after binding publication failures, and preserve uncertain reservations without redispatch. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
✅ Ponytail Reviewer completed successfully! Ponytail ReviewerOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅
|
|
✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft. Design Decision Gate 🏗️Outcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ Test Quality Sentinel completed test quality analysis. Test Quality SentinelOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ PR Code Quality Reviewer completed the code quality review.
|
Comment MemoryPeek at saved memory (pr-code-quality-reviewer)Note This comment is managed by comment memory.Expand the saved memory block to view or edit the persistent context for this thread.
|
There was a problem hiding this comment.
Request changes
The retry/recovery direction is sensible, but the implementation still binds launch verification to mutable run state and the new recovery test does not actually pin the retained-evidence path.
Themes
verifyLaunchedRun()still reads the latest workflow run instead of immutable attempt 1, so an early rerun can turn a valid accepted launch intorerun_not_authorizedduring the new retry window.- The regression test for "retain verified evidence after bound publish failure" can still pass via normal run discovery, so it doesn't prove the exact recovery mechanism this patch is introducing.
🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 53 AIC · ⌖ 5.97 AIC · ⊞ 21.7K
Comment /review to run again
There was a problem hiding this comment.
Skills-Based Review 🧠
Applied /diagnosing-bugs (regression coverage for the fixed race) and /codebase-design (shared vocabulary for the new retry/evidence-retention logic). No blocking issues.
📋 Key Themes & Highlights
Key Themes
- Root cause correctly scoped: the fix retries only transient native-run verification reads (404/5xx/timeout/pending-title-correlation) within the pool's existing
reconciliation.max_attempts/deadline_msbounds - it does not loosen any identity, revision, principal, attempt, or dispatch-token check, and still sends exactly one dispatch POST. - Evidence retention is sound:
verifiedEvidenceis captured only aftervalidateNativeRunsucceeds and is attached to theuncertain/binding_unconfirmedmarker so a laterreconcileDispatchcan recover the exact bound run without redispatch - and the replay validator (work_queue_replay.cjsassertEvidence) still re-checks scope/kind on that evidence before it can be trusted, so an unverified response can never masquerade as binding evidence. - Test coverage matches the stated risk surface: new tests cover each transient failure mode with exactly one retry + one POST, attempt/deadline exhaustion without releasing the reservation, binding-publication-failure recovery without redispatch, denied/mismatched runs failing fast with zero retries, and deadline-exact cutoff - this is a solid
/tdd-style regression suite for a fix whose original root cause (historical debug-gated exceptions) could not be reproduced retrospectively, per the PR's own investigation notes. - Minor maintainability nits only (see inline comments): the transient-error predicate could be a named, independently-testable helper rather than an inline boolean, and the 1s poll-interval cap benefits from a short comment distinguishing it from the unrelated exponential-backoff constant used in
work_queue_reconciler.cjs's own attempt loop.
Positive Highlights
- ✅ Still exactly one dispatch POST per launch attempt; retries are confined to read-only verification.
- ✅
log.failure(..., true)additions make verification/binding failures visible without requiringDEBUG, matching the stated goal. - ✅ Docs (
work-queue.md) updated in the same PR to describe the new retry/evidence-retention behavior. - ✅ Changeset included and scoped as a patch.
🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · 114.7 AIC · ⌖ 15.1 AIC · ⊞ 11.7K
Comment /matt to run again
There was a problem hiding this comment.
🔵 Needs a closer look
Safety-sensitive launch and ledger recovery need maintainer review, including the unresolved recovery gap after compaction.
1 open finding
What changed in this PR
Improves gh-aw work-queue launch recovery without dispatching workers twice.
Changes:
- Retries transient verification failures within existing policy limits.
- Retains verified evidence when binding publication fails and enables sanitized diagnostics.
- Adds recovery tests, documentation, and a patch changeset.
| File | Description |
|---|---|
| docs/src/content/docs/reference/work-queue.md | Documents retry and recovery guarantees. |
| actions/setup/js/work_queue_dispatch.test.cjs | Covers retries, recovery, and reservation retention. |
| actions/setup/js/work_queue_dispatch.cjs | Adds bounded verification retries and retained evidence. |
| .changeset/retain-accepted-queue-launch-identity.md | Records the patch release change. |
🧠 Review effort: Max
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Thanks for this focused fix, @pelikhan! The work-queue launch verification retry logic is well-tested and documented. This PR looks ready for review.
|
Verify immutable attempt one, retain recovery hint visibility across checkpoint history, isolate hint recovery tests, and name transient verification failures. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Exercise getWorkflowRunAttempt for attempt 1 in both selected-credential launch routes and fail explicitly if verification reads the latest-run endpoint. Preserve the single-POST, durable START, and principal binding assertions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
🎉 This pull request is included in a new release. Release: |

Summary
Investigation
The first failing boundary in dispatcher runs 38713409614, 38829188984, and 38845385623 is accepted-launch verification, not dispatch rejection. All nine workers were created at their dispatchers' pinned revisions:
For the latest dispatcher, authoritative ledger operations show START → binding_unconfirmed → trusted_activation binding for each original assignment. A local replay of the pre-failure ledger accepts the current exact native worker proof. Main at 7573428 still used a one-shot verification read and discarded verified evidence if binding publication failed.
Historical uncertainty: the original native read/validation/publication exceptions were debug-gated, so their exact causes cannot be established retrospectively. This PR fixes verified resilience/recovery gaps; it does not claim to prove that a particular transient HTTP error caused those runs. Existing reservations were inspected read-only, not repaired.
Validation
make fmt-cjsandmake lint-cjspass.PATH="/opt/homebrew/bin:$PATH" make agent-report-progresspasses, including typecheck, 126 impacted JavaScript tests, lint, schema checks, and the impacted Go selector (no Go changes).Limitations
No live rerun was authorized or performed. The policy deadline limits retry scheduling; already in-flight API requests retain their existing request timeouts.