Repository navigation
feat: add consuming shutdown for nodes using VSS - #131
ben-kaufman wants to merge 3 commits into
Conversation
talosmachina
left a comment
There was a problem hiding this comment.
1 finding (1 P1). Adds Node::try_into_shutdown, a consuming shutdown that drains background tasks, Electrum callbacks, peer workers, the final processor persist and VSS I/O before reporting success. Reviewed 92abaca, full tier.
What I checked, and 4 candidates I ruled out
Read in full: src/quiescence.rs, src/quiescence_tests.rs, the diffs of src/runtime.rs, src/lib.rs, src/connection.rs, src/chain/electrum.rs, src/chain/mod.rs, src/io/vss_store.rs, src/io/local_graph_store.rs, src/liquidity.rs, src/payment/*, src/builder.rs
Call sites traced: spawn_tracked_task, abort_cancellable_background_tasks, allow_task_spawns_with_timeout, WorkPermit::drop, NodeShutdown::wait
Build/tests: not run. The sandboxed runner could not confirm the author's write access on this repo and failed closed, so this review is static. No CI checks are reported on the branch.
Cold reader: the second-model check timed out, so the finding below was not independently challenged.
Ruled out
unified_qr_paymentbypasses the handle count: it is built fromonchain_payment(),bolt11_payment()andbolt12_payment(), each of which clonesshutdown_owner, so a retained QR handle still blocks entry.Electrum spawn_blockingpanics after closure:drain_native_callsruns only afterdrain_backgroundsucceeded, so no original caller can still be insidespawn_blocking; theexpectholds on the strict path.- Store drained before the processor's last write: phases run processor then store, and
close_and_draincloses VSS admission only after the processor task returned. - Late handshake reconnecting during drain:
close_admissionruns inbegin, sodo_connect_peerand inbound setup refuse newTrackedPeerManagers, and the repeateddisconnect_all_peerscovers an already admitted one.
Merge confidence: 3/5, one P1 (an ordinary stop permanently fails the later strict shutdown).
piotr-iohk
left a comment
There was a problem hiding this comment.
QA review
Scope: Full PR review at 92abaca, against merge base a265639, covering shutdown ownership, admission, native task drainage, final persistence and affected Rust/UniFFI contracts.
No new actionable code findings.
Existing concern: The ordinary-stop/strict-shutdown thread remains open. An independent loopback probe confirmed that start → stop → start → try_into_shutdown → wait returns Err(TaskFailed). Resolve the ordinary-cancellation accounting and retain a regression test for this sequence.
Validation: cargo test --locked --lib passed 143 tests; the UniFFI variant passed 140; documentation tests passed all 14; cargo check --locked --all-targets passed. The reconciliation probe used a separate checkout of the pinned head with one added test. External bitcoind/electrs, CLN, LND and real-server VSS integration suites were not executed; VSS drainage was exercised using the committed loopback fixtures.
Reviewer model: gpt-6.1-sol
Reasoning effort: high
There was a problem hiding this comment.
Verdict: ✅ Approve
Findings: None against the documented shutdown contract.
QA for 92abaca:
Native node and persistence lifecycle ✅ Workspace library tests passed.
Executed cargo test --locked --workspace --lib. The native fixtures exercised final processor persistence before VSS closure, retained child and draft refusal, same-owner timeout retries, sticky failures, canceled VSS waiters, Electrum callback lifetimes, peer workers and late handshakes.
Mobile binding feature compatibility ✅ Library tests passed with UniFFI enabled.
Executed cargo test --locked --lib --features uniffi. This checks the Rust library with its existing binding feature enabled; the consuming API has no mobile binding.
Shutdown API examples ✅ Documentation tests passed with default and UniFFI features.
Executed cargo test --locked --doc and cargo test --locked --doc --features uniffi, including both consuming shutdown examples.
Ordinary restart and sticky interruption ✅ Restart succeeded while later consuming waits retained cancellation failures.
Review-only probes used the existing loopback VSS fixture:
- All twelve unmodified
start→stop→starttrials restarted successfully; consuming shutdown then returnedTaskFailed. - A deterministic canceled idle task also allowed restart and retained
TaskFailedacross repeated consuming waits.
This confirms the mechanism in the existing review thread, also reproduced in the other QA review. The README explicitly retains interrupted-task failures; ordinary restartability does not promise later shutdown certification.
Check: all Rust targets compile ✅ All targets built with development panic unwinding enabled.
The normal cargo build --locked --workspace --lib passed. All targets also built with CARGO_PROFILE_DEV_PANIC=unwind cargo build --locked --workspace --all-targets after clearing the crate's build artifacts. The default all-target build hit duplicate library artifacts; Cargo documents this combination of panic=abort, cdylib and tests in rust-lang/cargo#6313. The manifest and lockfile are identical at the base and head. Compiler warnings remain.
Full review of a265639..92abaca, all 17 changed files and reached shutdown paths, with an independent reviewer. No mobile device run applies to this Rust-only API change; native fixtures exercised the changed behavior. External authenticated VSS deployment, CLN/LND, regtest integration suites and the unchanged payments benchmark were not rerun.
Reviewed by gpt-6.1-sol-high
piotr-iohk
left a comment
There was a problem hiding this comment.
Existing concern: The #131 (comment) remains open. An independent loopback probe confirmed that start → stop → start → try_into_shutdown → wait returns Err(TaskFailed). Resolve the ordinary-cancellation accounting and retain a regression test for this sequence.
piotr-iohk
left a comment
There was a problem hiding this comment.
Reconsidered this concern: the reproduced TaskFailed after ordinary stop/restart matches the documented sticky-failure contract, so I’m removing my blocker.
Nonblocking suggestion: explicitly document that ordinary stop/restart can prevent a later consuming shutdown from reporting success, and add a regression test covering that intended behavior.
piotr-iohk
left a comment
There was a problem hiding this comment.
QA review
Scope: Follow-up review at dc9aee1, covering all changes since 92abaca plus the affected cancellation, ordinary stop/restart and strict-shutdown paths. Unchanged surfaces are inherited from the baseline QA review. The merge base is unchanged at a265639, and the delta adds no surfaces beyond the four files it touches.
1 actionable finding — resolve or provide an evidence-backed rebuttal.
The ordinary-stop/strict-shutdown thread is marked resolved. By source reading, the reorder fixes its idle-task case: a task that observes the stop signal now completes its permit, and cancelling a still-pending future stays a failure. Sticky failure for genuinely interrupted work is the accepted contract, so this review does not re-report it. The one remaining case is the inline finding: the inbound-peer wrapper is only a waiter, so cancelling it interrupts no work.
The non-blocking suggestion to document that ordinary stop/restart can prevent a later consuming shutdown from reporting success is still open. The README and API docs did not change in this delta.
Validation: Nothing was compiled or executed in this review, because no Rust toolchain was available and the rustup and crates.io hosts were denied. All conclusions are source analysis. Dependency behavior was read from GitHub tags rather than checksummed crates: lightning-net-tokio 0.2.0 at c7583f3, and lightning 0.2.7 with lightning-background-processor 0.2.3 at b48de38. This head has no hosted check runs. The author reports passing local runs; those are not results from this review. The qa_contract.py preflight was not available in this workspace, so this structured result was not validated by it.
Evidence gap: settle_closed_channel_balances in tests/reorg_test.rs polls for about 10 seconds without re-syncing. The sweeper only regenerates spends at the end of sync_wallets or on the 30-second background-processor timer, so a sweep tracked just after that sync could exhaust the poll. This timing was not demonstrated and is not reported as a defect; a hosted reorg_test run would settle it.
Suggested additional test cases
- Rust loopback VSS fixture. Setup: start a VSS node with a listening address and an owned runtime, then connect a second node to it and wait until the peer appears on the listener. Action: call
stop(),start(),try_into_shutdown(), thenwait()on the listening node. Expected:Ok(()), or an explicitly documented refusal if a live inbound peer is meant to prevent certification.
Findings
- [LOW] Ordinary stop with a live inbound peer still fails later strict shutdown — inline at
src/lib.rs:538.
Reviewer model: claude-opus-5-5
Reasoning effort: high
There was a problem hiding this comment.
Verdict: ✅ Approve
Findings: No new findings against the documented shutdown contract. The existing inbound-peer waiter limitation is confirmed below: ordinary stop can still prevent a later consuming shutdown from certifying success.
QA for dc9aee1:
Completed tasks and pending cancellation ✅ Native lifecycle tests passed, including ordinary restart followed by consuming shutdown.
Executed CARGO_PROFILE_DEV_PANIC=unwind cargo test --locked --workspace --lib at the unchanged head. The updated stop/restart regression succeeds and repeats its wait without replay. The deterministic queued control verifies that actual Ready completion succeeds while cancelling Pending work retains TaskFailed across repeated waits. Existing persistence, panic, timeout and native-worker controls also pass.
Cancellation with mobile binding features ✅ The lifecycle tests also passed with UniFFI enabled.
Executed CARGO_PROFILE_DEV_PANIC=unwind cargo test --locked --lib --features uniffi at the unchanged head, including the same Ready/Pending and ordinary-restart controls. This checks Rust feature compatibility; the consuming API has no mobile binding.
Live inbound peers and sticky cancellation ✅ A native probe confirmed direct drainage succeeds and ordinary-stop cancellation remains sticky.
A review-only probe reused the committed loopback VSS fixture and waited for an actual inbound handshake before each sequence:
| Listening node sequence | First consuming wait | Repeated wait |
|---|---|---|
| Direct consuming shutdown | Ok(()) |
Ok(()) |
| Ordinary stop, then consuming shutdown | Err(TaskFailed) |
Err(TaskFailed) |
| Ordinary stop, restart, then consuming shutdown | Err(TaskFailed) |
Err(TaskFailed) |
Restart itself succeeded. This adds runtime evidence to the existing thread. The wrapper is still cancelled before reporting completion, which fits the documented sticky-cancellation contract; the result is a conservative refusal, not proof that original native work remains active. The probe was removed after execution.
Reorganization settlement and saved deep regression ✅ The saved regression and randomized reorg property passed.
Executed CARGO_PROFILE_DEV_PANIC=unwind cargo test --locked --test reorg_test. The persisted six-block/cooperative-flag regression runs before randomized cases. Final spendable-balance, anchor-reserve and empty-Lightning assertions remain enforced. The sweep-poll timing concern in the source review did not reproduce in this run.
The native fixture used Bitcoin Core 27.2 and the pinned Electrs source, with build compatibility patches for ARM64 and current compiler tooling. The PR sources were unchanged.
Reorganization settlement with mobile binding features ✅ The reorg property also passed with UniFFI enabled.
Executed CARGO_PROFILE_DEV_PANIC=unwind cargo test --locked --test reorg_test --features uniffi against the same native fixture, including the saved regression and final balance assertions.
Delta review of 92abaca..dc9aee1: all four changed files, reached cancellation and shutdown paths, and prior review concerns, with an independent reviewer. Unchanged documentation examples and other unchanged surfaces rely on the previous passing review.
No mobile device run applies to this Rust-only source API; native fixtures exercised the changed behavior. Other regtest suites, external authenticated VSS, CLN/LND and the unchanged payments benchmark were not rerun. Compiler warnings remain.
Reviewed by gpt-6.1-sol-high
piotr-iohk
left a comment
There was a problem hiding this comment.
QA review
Scope: follow-up review at d1f9b48 of the entire delta since dc9aee1, including affected inbound-worker and shutdown paths. Unchanged coverage is inherited from the completed baseline; ancestry, base and merge base were verified locally.
No new actionable code findings.
Validation: On Linux with Rust 1.90.0, all 13 quiescence tests and 16 runtime tests passed, including the new inbound stop/restart/shutdown regression. Dependency analysis used Cargo.lock-verified sources: lightning-net-tokio 0.2.0 (SHA-256 8055737e3d2d06240a3fdf10e26b2716110fcea90011a0839e8e82fc6e58ff5e) and lightning 0.2.7 (SHA-256 982ffb82e8362ea95e03a5de8a6004bceb7a577fc1f07d5b02cfc3fc7c930a90).
Execution covered the focused shutdown/runtime suites; full all-target and UniFFI suites were not rerun. Earlier findings and discussion were intentionally left to the separate reconciliation stage.
Earlier findings reconciled at this revision: 3 verified fixed.
Reviewer model: gpt-6.1-sol
Reasoning effort: high
There was a problem hiding this comment.
Verdict: ✅ Approve
Findings: None. The inbound-peer waiter concern is fixed: registration completes without waiting for disconnection, while the actual read and event workers retain their native tracking lease. Genuine pending cancellation remains a sticky failure.
QA for d1f9b48:
Inbound shutdown with mobile binding features ✅ The UniFFI library suite passed, including successful consuming shutdown after an inbound connection, ordinary stop and restart.
Executed CARGO_PROFILE_DEV_PANIC=unwind cargo test --locked --lib --features uniffi against the unchanged head. The committed regression waits for a real inbound handshake and verifies both the initial consuming wait and repeated zero-timeout wait. Late-handshake drainage, retained peer-worker/panic accounting and Ready-versus-Pending cancellation controls also passed.
Shutdown API examples ✅ Default documentation tests passed, including both consuming-shutdown examples.
Executed CARGO_PROFILE_DEV_PANIC=unwind cargo test --locked --doc. Both consuming-shutdown examples are compile-only; this does not claim they were executed.
Shutdown API examples with mobile binding features ✅ Documentation tests also passed with UniFFI enabled.
Executed CARGO_PROFILE_DEV_PANIC=unwind cargo test --locked --doc --features uniffi, including the same compile-only consuming-shutdown examples.
Default runtime and quiescence suites already passed at this exact head in the current-head QA review. The unchanged default and UniFFI reorganization properties retain the passing results in the previous review.
Delta review of dc9aee1..d1f9b48: all three changed files, affected inbound-worker and shutdown paths, and prior concerns, with an independent reviewer. The updated cancellation documentation matches the retained failure controls.
No mobile device run applies to this Rust-only API, which has no mobile binding. Native fixtures exercised the changed behavior. Other regtest suites, external authenticated VSS, CLN/LND and the payments benchmark were not rerun in this delta. Compiler warnings remain.
Reviewed by gpt-6.1-sol-high
Ordinary stop can finish while canceled waiters leave native work running. This adds a consuming Rust API,
Node::try_into_shutdown, which retains the original node while background tasks, Electrum callbacks, peer workers, final processor persistence and original VSS I/O drain. A timeout retains the same shutdown owner; persistence failures, panics and canceled native work cannot become success on a later wait.Entry refuses outstanding payment/liquidity/prepared-send handles, unsupported stores and borrowed runtimes without changing the original node. Ordinary stop/restart remains available. The final peer disconnect also covers a handshake that completes during background drainage. When task completion and cancellation are both ready, a returned
Readyresult completes its permit first; cancellation of a still-pending task remains a sticky failure. The real stop/restart/shutdown regression and deterministic ready-versus-pending controls cover this distinction. Connected inbound peers retain their own tracked worker permits; stopping a join-only waiter no longer makes a later consuming shutdown fail. A two-node regression covers ordinary stop, restart and repeated shutdown waits after a real inbound handshake.This is Rust source preparation for original-node shutdown. It adds no mobile binding, dependency/version change, release or server activation. A successful drain covers this node only: it is neither a complete-wallet backup nor proof that independently constructed writers or the other original wallet domain are fenced.
Local validation on all 168 frozen final inputs:
os error 48) before its assertions. Its 307 passing cases are retained; one bounded exact-case recovery passed, followed by the previously unrun reorg case. The failed attempt remains recorded. Feature and explicit default documentation runs each passed 14 cases, including 13 compile-only examples.The required UniFFI gate exposed an existing reorg fixture defect. The saved six-block/cooperative-flag case failed identically on the published parent and actual main: funding unconfirmation caused a force close whose outputs had not matured. The corrected fixture settles the actual claimable/sweep outputs and retains cooperative-reason checks where applicable, all balance/reserve/empty-Lightning assertions, all depths and the original regression seed. Fixed shallow and deep cases passed before the complete final runs.
Historical failures remain recorded: the initial x86 Electrs fixture, a default-development target collision, the old reorg assertion and a sandbox socket-denial attempt. Final native gates use the existing arm64 Electrs, isolated regtest data and explicit
CARGO_PROFILE_DEV_PANIC=unwind; the socket recovery changed permission only. No unchanged hosted job was retried. CLN, LND and external-VSS targets are cfg-disabled, not runtime passes. No measured 100% coverage, released FFI or authenticated mobile acceptance is claimed.All three commits in the current main-to-head range are cryptographically verified locally and by GitHub. No hosted workflow run has been triggered for this head; the local results above are not hosted CI results.