Skip to content

feat: add consuming shutdown for nodes using VSS - #131

Open
ben-kaufman wants to merge 3 commits into
mainfrom
codex/terminal-node-quiescence-20261105
Open

ben-kaufman wants to merge 3 commits into
mainfrom
codex/terminal-node-quiescence-20261105

Conversation

@ben-kaufman

@ben-kaufman ben-kaufman commented Nov 5, 2026 •

Copy link
Copy Markdown

Ordinary stop can finish while canceled waiters leave native work running. This adds a consuming Rust API, Node::try_into_shutdown, which retains the original node while background tasks, Electrum callbacks, peer workers, final processor persistence and original VSS I/O drain. A timeout retains the same shutdown owner; persistence failures, panics and canceled native work cannot become success on a later wait.

Entry refuses outstanding payment/liquidity/prepared-send handles, unsupported stores and borrowed runtimes without changing the original node. Ordinary stop/restart remains available. The final peer disconnect also covers a handshake that completes during background drainage. When task completion and cancellation are both ready, a returned Ready result completes its permit first; cancellation of a still-pending task remains a sticky failure. The real stop/restart/shutdown regression and deterministic ready-versus-pending controls cover this distinction. Connected inbound peers retain their own tracked worker permits; stopping a join-only waiter no longer makes a later consuming shutdown fail. A two-node regression covers ordinary stop, restart and repeated shutdown waits after a real inbound handshake.

This is Rust source preparation for original-node shutdown. It adds no mobile binding, dependency/version change, release or server activation. A successful drain covers this node only: it is neither a complete-wallet backup nor proof that independently constructed writers or the other original wallet domain are fenced.

Local validation on all 168 frozen final inputs:

  • Format, all-target Clippy and all-target build passed with no introduced findings. Full structured diagnostics retain existing formatter, build and UniFFI warnings. The complete final Clippy diagnostic objects and nested notes introduce no findings.
  • All-target tests: 314 distinct named cases passed, plus four intentional child-process executions; no failed or ignored cases.
  • UniFFI coverage: 309 distinct native cases eventually passed, plus four intentional child-process executions. The first command stopped after one test selected an occupied loopback port (os error 48) before its assertions. Its 307 passing cases are retained; one bounded exact-case recovery passed, followed by the previously unrun reorg case. The failed attempt remains recorded. Feature and explicit default documentation runs each passed 14 cases, including 13 compile-only examples.
  • The unchanged Criterion test-mode control reached 1,000 successful payment events after 131 failed events and 13 immediate-send refusals. These are eventual outcomes, not first-attempt successes or performance measurements; the final return payment is checked for dispatch, not independently acknowledged settlement.
  • Two independent production source reviews, the affected fixture review and two independent reviews of the connected-inbound correction are clean. Controlled tests retain original ownership on timeout, pending handles, cancellation and panic, and cover native callback/peer-worker lifetimes. The earlier late-handshake negative control remains preserved.

The required UniFFI gate exposed an existing reorg fixture defect. The saved six-block/cooperative-flag case failed identically on the published parent and actual main: funding unconfirmation caused a force close whose outputs had not matured. The corrected fixture settles the actual claimable/sweep outputs and retains cooperative-reason checks where applicable, all balance/reserve/empty-Lightning assertions, all depths and the original regression seed. Fixed shallow and deep cases passed before the complete final runs.

Historical failures remain recorded: the initial x86 Electrs fixture, a default-development target collision, the old reorg assertion and a sandbox socket-denial attempt. Final native gates use the existing arm64 Electrs, isolated regtest data and explicit CARGO_PROFILE_DEV_PANIC=unwind; the socket recovery changed permission only. No unchanged hosted job was retried. CLN, LND and external-VSS targets are cfg-disabled, not runtime passes. No measured 100% coverage, released FFI or authenticated mobile acceptance is claimed.

All three commits in the current main-to-head range are cryptographically verified locally and by GitHub. No hosted workflow run has been triggered for this head; the local results above are not hosted CI results.

@talosmachina talosmachina left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 finding (1 P1). Adds Node::try_into_shutdown, a consuming shutdown that drains background tasks, Electrum callbacks, peer workers, the final processor persist and VSS I/O before reporting success. Reviewed 92abaca, full tier.

What I checked, and 4 candidates I ruled out

Read in full: src/quiescence.rs, src/quiescence_tests.rs, the diffs of src/runtime.rs, src/lib.rs, src/connection.rs, src/chain/electrum.rs, src/chain/mod.rs, src/io/vss_store.rs, src/io/local_graph_store.rs, src/liquidity.rs, src/payment/*, src/builder.rs
Call sites traced: spawn_tracked_task, abort_cancellable_background_tasks, allow_task_spawns_with_timeout, WorkPermit::drop, NodeShutdown::wait
Build/tests: not run. The sandboxed runner could not confirm the author's write access on this repo and failed closed, so this review is static. No CI checks are reported on the branch.
Cold reader: the second-model check timed out, so the finding below was not independently challenged.

Ruled out

  • unified_qr_payment bypasses the handle count: it is built from onchain_payment(), bolt11_payment() and bolt12_payment(), each of which clones shutdown_owner, so a retained QR handle still blocks entry.
  • Electrum spawn_blocking panics after closure: drain_native_calls runs only after drain_background succeeded, so no original caller can still be inside spawn_blocking; the expect holds on the strict path.
  • Store drained before the processor's last write: phases run processor then store, and close_and_drain closes VSS admission only after the processor task returned.
  • Late handshake reconnecting during drain: close_admission runs in begin, so do_connect_peer and inbound setup refuse new TrackedPeerManagers, and the repeated disconnect_all_peers covers an already admitted one.

Merge confidence: 3/5, one P1 (an ordinary stop permanently fails the later strict shutdown).

Comment thread src/runtime.rs Resolved

@piotr-iohk piotr-iohk left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QA review

Scope: Full PR review at 92abaca, against merge base a265639, covering shutdown ownership, admission, native task drainage, final persistence and affected Rust/UniFFI contracts.

No new actionable code findings.

Existing concern: The ordinary-stop/strict-shutdown thread remains open. An independent loopback probe confirmed that start → stop → start → try_into_shutdown → wait returns Err(TaskFailed). Resolve the ordinary-cancellation accounting and retain a regression test for this sequence.

Validation: cargo test --locked --lib passed 143 tests; the UniFFI variant passed 140; documentation tests passed all 14; cargo check --locked --all-targets passed. The reconciliation probe used a separate checkout of the pinned head with one added test. External bitcoind/electrs, CLN, LND and real-server VSS integration suites were not executed; VSS drainage was exercised using the committed loopback fixtures.

Reviewer model: gpt-6.1-sol
Reasoning effort: high

@ovi-reviewer ovi-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: ✅ Approve

Findings: None against the documented shutdown contract.

QA for 92abaca:

Native node and persistence lifecycle ✅ Workspace library tests passed.

Executed cargo test --locked --workspace --lib. The native fixtures exercised final processor persistence before VSS closure, retained child and draft refusal, same-owner timeout retries, sticky failures, canceled VSS waiters, Electrum callback lifetimes, peer workers and late handshakes.

Mobile binding feature compatibility ✅ Library tests passed with UniFFI enabled.

Executed cargo test --locked --lib --features uniffi. This checks the Rust library with its existing binding feature enabled; the consuming API has no mobile binding.

Shutdown API examples ✅ Documentation tests passed with default and UniFFI features.

Executed cargo test --locked --doc and cargo test --locked --doc --features uniffi, including both consuming shutdown examples.

Ordinary restart and sticky interruption ✅ Restart succeeded while later consuming waits retained cancellation failures.

Review-only probes used the existing loopback VSS fixture:

  • All twelve unmodified start → stop → start trials restarted successfully; consuming shutdown then returned TaskFailed.
  • A deterministic canceled idle task also allowed restart and retained TaskFailed across repeated consuming waits.

This confirms the mechanism in the existing review thread, also reproduced in the other QA review. The README explicitly retains interrupted-task failures; ordinary restartability does not promise later shutdown certification.

Check: all Rust targets compile ✅ All targets built with development panic unwinding enabled.

The normal cargo build --locked --workspace --lib passed. All targets also built with CARGO_PROFILE_DEV_PANIC=unwind cargo build --locked --workspace --all-targets after clearing the crate's build artifacts. The default all-target build hit duplicate library artifacts; Cargo documents this combination of panic=abort, cdylib and tests in rust-lang/cargo#6313. The manifest and lockfile are identical at the base and head. Compiler warnings remain.

Full review of a265639..92abaca, all 17 changed files and reached shutdown paths, with an independent reviewer. No mobile device run applies to this Rust-only API change; native fixtures exercised the changed behavior. External authenticated VSS deployment, CLN/LND, regtest integration suites and the unchanged payments benchmark were not rerun.


Reviewed by gpt-6.1-sol-high

@ovitrif ovitrif left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utAck

@piotr-iohk piotr-iohk left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Existing concern: The #131 (comment) remains open. An independent loopback probe confirmed that start → stop → start → try_into_shutdown → wait returns Err(TaskFailed). Resolve the ordinary-cancellation accounting and retain a regression test for this sequence.

@piotr-iohk piotr-iohk left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reconsidered this concern: the reproduced TaskFailed after ordinary stop/restart matches the documented sticky-failure contract, so I’m removing my blocker.

Nonblocking suggestion: explicitly document that ordinary stop/restart can prevent a later consuming shutdown from reporting success, and add a regression test covering that intended behavior.

@piotr-iohk piotr-iohk left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QA review

Scope: Follow-up review at dc9aee1, covering all changes since 92abaca plus the affected cancellation, ordinary stop/restart and strict-shutdown paths. Unchanged surfaces are inherited from the baseline QA review. The merge base is unchanged at a265639, and the delta adds no surfaces beyond the four files it touches.

1 actionable finding — resolve or provide an evidence-backed rebuttal.

The ordinary-stop/strict-shutdown thread is marked resolved. By source reading, the reorder fixes its idle-task case: a task that observes the stop signal now completes its permit, and cancelling a still-pending future stays a failure. Sticky failure for genuinely interrupted work is the accepted contract, so this review does not re-report it. The one remaining case is the inline finding: the inbound-peer wrapper is only a waiter, so cancelling it interrupts no work.

The non-blocking suggestion to document that ordinary stop/restart can prevent a later consuming shutdown from reporting success is still open. The README and API docs did not change in this delta.

Validation: Nothing was compiled or executed in this review, because no Rust toolchain was available and the rustup and crates.io hosts were denied. All conclusions are source analysis. Dependency behavior was read from GitHub tags rather than checksummed crates: lightning-net-tokio 0.2.0 at c7583f3, and lightning 0.2.7 with lightning-background-processor 0.2.3 at b48de38. This head has no hosted check runs. The author reports passing local runs; those are not results from this review. The qa_contract.py preflight was not available in this workspace, so this structured result was not validated by it.

Evidence gap: settle_closed_channel_balances in tests/reorg_test.rs polls for about 10 seconds without re-syncing. The sweeper only regenerates spends at the end of sync_wallets or on the 30-second background-processor timer, so a sweep tracked just after that sync could exhaust the poll. This timing was not demonstrated and is not reported as a defect; a hosted reorg_test run would settle it.

Suggested additional test cases

  • Rust loopback VSS fixture. Setup: start a VSS node with a listening address and an owned runtime, then connect a second node to it and wait until the peer appears on the listener. Action: call stop(), start(), try_into_shutdown(), then wait() on the listening node. Expected: Ok(()), or an explicitly documented refusal if a live inbound peer is meant to prevent certification.

Findings

  • [LOW] Ordinary stop with a live inbound peer still fails later strict shutdown — inline at src/lib.rs:538.

Reviewer model: claude-opus-5-5
Reasoning effort: high

Comment thread src/lib.rs Resolved

@ovi-reviewer ovi-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: ✅ Approve

Findings: No new findings against the documented shutdown contract. The existing inbound-peer waiter limitation is confirmed below: ordinary stop can still prevent a later consuming shutdown from certifying success.

QA for dc9aee1:

Completed tasks and pending cancellation ✅ Native lifecycle tests passed, including ordinary restart followed by consuming shutdown.

Executed CARGO_PROFILE_DEV_PANIC=unwind cargo test --locked --workspace --lib at the unchanged head. The updated stop/restart regression succeeds and repeats its wait without replay. The deterministic queued control verifies that actual Ready completion succeeds while cancelling Pending work retains TaskFailed across repeated waits. Existing persistence, panic, timeout and native-worker controls also pass.

Cancellation with mobile binding features ✅ The lifecycle tests also passed with UniFFI enabled.

Executed CARGO_PROFILE_DEV_PANIC=unwind cargo test --locked --lib --features uniffi at the unchanged head, including the same Ready/Pending and ordinary-restart controls. This checks Rust feature compatibility; the consuming API has no mobile binding.

Live inbound peers and sticky cancellation ✅ A native probe confirmed direct drainage succeeds and ordinary-stop cancellation remains sticky.

A review-only probe reused the committed loopback VSS fixture and waited for an actual inbound handshake before each sequence:

Listening node sequence First consuming wait Repeated wait
Direct consuming shutdown Ok(()) Ok(())
Ordinary stop, then consuming shutdown Err(TaskFailed) Err(TaskFailed)
Ordinary stop, restart, then consuming shutdown Err(TaskFailed) Err(TaskFailed)

Restart itself succeeded. This adds runtime evidence to the existing thread. The wrapper is still cancelled before reporting completion, which fits the documented sticky-cancellation contract; the result is a conservative refusal, not proof that original native work remains active. The probe was removed after execution.

Reorganization settlement and saved deep regression ✅ The saved regression and randomized reorg property passed.

Executed CARGO_PROFILE_DEV_PANIC=unwind cargo test --locked --test reorg_test. The persisted six-block/cooperative-flag regression runs before randomized cases. Final spendable-balance, anchor-reserve and empty-Lightning assertions remain enforced. The sweep-poll timing concern in the source review did not reproduce in this run.

The native fixture used Bitcoin Core 27.2 and the pinned Electrs source, with build compatibility patches for ARM64 and current compiler tooling. The PR sources were unchanged.

Reorganization settlement with mobile binding features ✅ The reorg property also passed with UniFFI enabled.

Executed CARGO_PROFILE_DEV_PANIC=unwind cargo test --locked --test reorg_test --features uniffi against the same native fixture, including the saved regression and final balance assertions.

Delta review of 92abaca..dc9aee1: all four changed files, reached cancellation and shutdown paths, and prior review concerns, with an independent reviewer. Unchanged documentation examples and other unchanged surfaces rely on the previous passing review.

No mobile device run applies to this Rust-only source API; native fixtures exercised the changed behavior. Other regtest suites, external authenticated VSS, CLN/LND and the unchanged payments benchmark were not rerun. Compiler warnings remain.


Reviewed by gpt-6.1-sol-high

@ovitrif ovitrif left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utAck

@piotr-iohk piotr-iohk left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QA review

Scope: follow-up review at d1f9b48 of the entire delta since dc9aee1, including affected inbound-worker and shutdown paths. Unchanged coverage is inherited from the completed baseline; ancestry, base and merge base were verified locally.

No new actionable code findings.

Validation: On Linux with Rust 1.90.0, all 13 quiescence tests and 16 runtime tests passed, including the new inbound stop/restart/shutdown regression. Dependency analysis used Cargo.lock-verified sources: lightning-net-tokio 0.2.0 (SHA-256 8055737e3d2d06240a3fdf10e26b2716110fcea90011a0839e8e82fc6e58ff5e) and lightning 0.2.7 (SHA-256 982ffb82e8362ea95e03a5de8a6004bceb7a577fc1f07d5b02cfc3fc7c930a90).

Execution covered the focused shutdown/runtime suites; full all-target and UniFFI suites were not rerun. Earlier findings and discussion were intentionally left to the separate reconciliation stage.

Earlier findings reconciled at this revision: 3 verified fixed.

Reviewer model: gpt-6.1-sol
Reasoning effort: high

@ovi-reviewer ovi-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: ✅ Approve

Findings: None. The inbound-peer waiter concern is fixed: registration completes without waiting for disconnection, while the actual read and event workers retain their native tracking lease. Genuine pending cancellation remains a sticky failure.

QA for d1f9b48:

Inbound shutdown with mobile binding features ✅ The UniFFI library suite passed, including successful consuming shutdown after an inbound connection, ordinary stop and restart.

Executed CARGO_PROFILE_DEV_PANIC=unwind cargo test --locked --lib --features uniffi against the unchanged head. The committed regression waits for a real inbound handshake and verifies both the initial consuming wait and repeated zero-timeout wait. Late-handshake drainage, retained peer-worker/panic accounting and Ready-versus-Pending cancellation controls also passed.

Shutdown API examples ✅ Default documentation tests passed, including both consuming-shutdown examples.

Executed CARGO_PROFILE_DEV_PANIC=unwind cargo test --locked --doc. Both consuming-shutdown examples are compile-only; this does not claim they were executed.

Shutdown API examples with mobile binding features ✅ Documentation tests also passed with UniFFI enabled.

Executed CARGO_PROFILE_DEV_PANIC=unwind cargo test --locked --doc --features uniffi, including the same compile-only consuming-shutdown examples.

Default runtime and quiescence suites already passed at this exact head in the current-head QA review. The unchanged default and UniFFI reorganization properties retain the passing results in the previous review.

Delta review of dc9aee1..d1f9b48: all three changed files, affected inbound-worker and shutdown paths, and prior concerns, with an independent reviewer. The updated cancellation documentation matches the retained failure controls.

No mobile device run applies to this Rust-only API, which has no mobile binding. Native fixtures exercised the changed behavior. Other regtest suites, external authenticated VSS, CLN/LND and the payments benchmark were not rerun in this delta. Compiler warnings remain.


Reviewed by gpt-6.1-sol-high

@ovitrif ovitrif left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utAck

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants