Skip to content

fix: retry accepted work-queue launch verification safely - #69230

Merged
pelikhan merged 4 commits into
mainfrom
pelikhan-sturdy-enigma
Oct 16, 2026
Merged

pelikhan merged 4 commits into
mainfrom
pelikhan-sturdy-enigma

Conversation

@pelikhan

Copy link
Copy Markdown
Collaborator

Summary

  • Retry only transient native run verification reads (404, 5xx, selected transport errors, pending dispatch-token title correlation), bounded by the pool's existing reconciliation attempt/deadline limits.
  • Keep every exact repository/workflow/SHA/principal/attempt/token check, send only one dispatch POST, and retain uncertain reservations without release or redispatch.
  • Preserve independently verified native evidence when subsequent binding publication fails, so existing reconciliation can recover the exact run. Never assert native evidence for an unverified response.
  • Emit sanitized verification/binding diagnostics without requiring DEBUG; document the behavior and add a patch changeset.

Investigation

The first failing boundary in dispatcher runs 38713409614, 38829188984, and 38845385623 is accepted-launch verification, not dispatch rejection. All nine workers were created at their dispatchers' pinned revisions:

Dispatcher Existing worker runs SHA
38713409614 38715064209, 38715093502, 38715121823 5eb4c29
38829188984 38830686838, 38830708106, 38830731349 c413774
38845385623 38845828756, 38845844758, 38845860941 573f10f

For the latest dispatcher, authoritative ledger operations show START → binding_unconfirmed → trusted_activation binding for each original assignment. A local replay of the pre-failure ledger accepts the current exact native worker proof. Main at 7573428 still used a one-shot verification read and discarded verified evidence if binding publication failed.

Historical uncertainty: the original native read/validation/publication exceptions were debug-gated, so their exact causes cannot be established retrospectively. This PR fixes verified resilience/recovery gaps; it does not claim to prove that a particular transient HTTP error caused those runs. Existing reservations were inspected read-only, not repaired.

Validation

  • 194 focused fixture tests pass across dispatch, native validation, binding, and control receipts.
  • make fmt-cjs and make lint-cjs pass.
  • PATH="/opt/homebrew/bin:$PATH" make agent-report-progress passes, including typecheck, 126 impacted JavaScript tests, lint, schema checks, and the impacted Go selector (no Go changes).
  • Regressions cover transient read recovery, attempt/deadline exhaustion, binding publication recovery, denied/mismatched native identities, one POST only, and reservation retention.
  • No workflow dispatches, reruns, live queue mutations, queue resets, or reservation releases were performed. No workflow source/lock files or health code changed.

Limitations

No live rerun was authorized or performed. The policy deadline limits retry scheduling; already in-flight API requests retain their existing request timeouts.

Bound transient verification reads by the existing reconciliation policy, retain independently verified evidence after binding publication failures, and preserve uncertain reservations without redispatch.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pelikhan
pelikhan marked this pull request as ready for review October 16, 2026 04:29
Copilot AI max review requested due to automatic review settings October 16, 2026 04:29
@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Ponytail Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Ponytail Reviewer · gpt56 · 6.33 AIC · ⌖ 5.35 AIC · ⊞ 14.6K · ◷

Generated by Ponytail Reviewer for #69230

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Design Decision Gate 🏗️

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Design Decision Gate 🏗️ · opus50 · 14 AIC · ⌖ 45.9 AIC · ◇ 28.1 AIC · ⊞ 1.9K · ◷

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · haiku45 · 5.16 AIC · ⌖ 7.96 AIC · ◇ 2.53 AIC · ⊞ 9.2K · ◷

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

Copy link
Copy Markdown
Contributor

Comment Memory

Peek at saved memory (pr-code-quality-reviewer)
reviewed_at: 2026-10-16T04:32:53Z
review_event: REQUEST_CHANGES
top_themes:
- launch verification reads mutable latest run instead of immutable attempt 1
- recovery test does not isolate retained-evidence hint path
files_reviewed:
- actions/setup/js/work_queue_dispatch.cjs
- actions/setup/js/work_queue_dispatch.test.cjs
- docs/src/content/docs/reference/work-queue.md
comment_count: 2

Note

This comment is managed by comment memory.

Expand the saved memory block to view or edit the persistent context for this thread.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 53 AIC · ⌖ 5.97 AIC · ⊞ 21.7K · ◷
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes

The retry/recovery direction is sensible, but the implementation still binds launch verification to mutable run state and the new recovery test does not actually pin the retained-evidence path.

Themes
  • verifyLaunchedRun() still reads the latest workflow run instead of immutable attempt 1, so an early rerun can turn a valid accepted launch into rerun_not_authorized during the new retry window.
  • The regression test for "retain verified evidence after bound publish failure" can still pass via normal run discovery, so it doesn't prove the exact recovery mechanism this patch is introducing.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 53 AIC · ⌖ 5.97 AIC · ⊞ 21.7K
Comment /review to run again

Comment thread actions/setup/js/work_queue_dispatch.cjs Outdated
Comment thread actions/setup/js/work_queue_dispatch.test.cjs

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /diagnosing-bugs (regression coverage for the fixed race) and /codebase-design (shared vocabulary for the new retry/evidence-retention logic). No blocking issues.

📋 Key Themes & Highlights

Key Themes

  • Root cause correctly scoped: the fix retries only transient native-run verification reads (404/5xx/timeout/pending-title-correlation) within the pool's existing reconciliation.max_attempts/deadline_ms bounds - it does not loosen any identity, revision, principal, attempt, or dispatch-token check, and still sends exactly one dispatch POST.
  • Evidence retention is sound: verifiedEvidence is captured only after validateNativeRun succeeds and is attached to the uncertain/binding_unconfirmed marker so a later reconcileDispatch can recover the exact bound run without redispatch - and the replay validator (work_queue_replay.cjs assertEvidence) still re-checks scope/kind on that evidence before it can be trusted, so an unverified response can never masquerade as binding evidence.
  • Test coverage matches the stated risk surface: new tests cover each transient failure mode with exactly one retry + one POST, attempt/deadline exhaustion without releasing the reservation, binding-publication-failure recovery without redispatch, denied/mismatched runs failing fast with zero retries, and deadline-exact cutoff - this is a solid /tdd-style regression suite for a fix whose original root cause (historical debug-gated exceptions) could not be reproduced retrospectively, per the PR's own investigation notes.
  • Minor maintainability nits only (see inline comments): the transient-error predicate could be a named, independently-testable helper rather than an inline boolean, and the 1s poll-interval cap benefits from a short comment distinguishing it from the unrelated exponential-backoff constant used in work_queue_reconciler.cjs's own attempt loop.

Positive Highlights

  • ✅ Still exactly one dispatch POST per launch attempt; retries are confined to read-only verification.
  • ✅ log.failure(..., true) additions make verification/binding failures visible without requiring DEBUG, matching the stated goal.
  • ✅ Docs (work-queue.md) updated in the same PR to describe the new retry/evidence-retention behavior.
  • ✅ Changeset included and scoped as a patch.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · 114.7 AIC · ⌖ 15.1 AIC · ⊞ 11.7K
Comment /matt to run again

Comment thread actions/setup/js/work_queue_dispatch.cjs Outdated
Comment thread actions/setup/js/work_queue_dispatch.cjs

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Safety-sensitive launch and ledger recovery need maintainer review, including the unresolved recovery gap after compaction.

1 open finding
What changed in this PR

Improves gh-aw work-queue launch recovery without dispatching workers twice.

Changes:

  • Retries transient verification failures within existing policy limits.
  • Retains verified evidence when binding publication fails and enables sanitized diagnostics.
  • Adds recovery tests, documentation, and a patch changeset.
File Description
docs/​src/​content/​docs/​reference/​work-queue.md Documents retry and recovery guarantees.
actions/​setup/​js/​work_queue_dispatch.test.cjs Covers retries, recovery, and reservation retention.
actions/​setup/​js/​work_queue_dispatch.cjs Adds bounded verification retries and retained evidence.
.changeset/​retain-accepted-queue-launch-identity.md Records the patch release change.

🧠 Review effort: Max


💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Comment thread actions/setup/js/work_queue_dispatch.cjs
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for this focused fix, @pelikhan! The work-queue launch verification retry logic is well-tested and documented. This PR looks ready for review.

Generated by ✅ Contribution Check · copilot · auto · 35.6 AIC · ⌖ 0.637 AIC · ⊞ 10.6K · ◷

pelikhan and others added 2 commits October 15, 2026 22:02
Verify immutable attempt one, retain recovery hint visibility across checkpoint history, isolate hint recovery tests, and name transient verification failures.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Exercise getWorkflowRunAttempt for attempt 1 in both selected-credential launch routes and fail explicitly if verification reads the latest-run endpoint. Preserve the single-POST, durable START, and principal binding assertions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pelikhan
pelikhan merged commit f55dc3a into main Oct 16, 2026
16 of 18 checks passed
@pelikhan
pelikhan deleted the pelikhan-sturdy-enigma branch October 16, 2026 05:22
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v0.92.3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants