[FrostGod]

Summary

After SIGTERM, openshell-sandbox could stay alive until the container runtime SIGKILLed it, so every Kubernetes sandbox delete waited out the pod's 30 s termination grace period. The boundary now stops its process runtime within two seconds after it stops serving.

Related Issue

Part of #5025. This PR fixes the openshell-sandbox half; #5025 stays open for the supervisor pod, which can still wait out the grace period.

Changes

• run_boundary serves through serve_then_stop, which calls shutdown_timeout with a 2 s bound after serve returns, on every return path. Dropping the runtime used to wait for blocking tasks with no limit.
• Workload kill and reaping are unchanged: termination still SIGKILLs the registered process groups before serve returns.
• Added sigterm_exits_while_the_supervisor_holds_an_exchange_open. It runs the boundary in a child process behind a fake supervisor that starts an agent and keeps a Wait and an AcceptNetwork exchange open, sends SIGTERM, and requires an exit within 10 s.

Root cause: the supervisor keeps an agent Wait exchange open for the life of the sandbox. ManagedProcess::wait blocks on a condition variable that only the agent's async reaper task wakes. When the runtime shutdown cancelled the reaper before it recorded the exit, Wait never returned, and the main thread stayed in BlockingPool::shutdown under drop(Runtime) until SIGKILL. gdb on a hung run showed the only other runtime thread in serve_one → dispatch → BoundaryRuntime::wait → ManagedProcess::wait.

Release Notes

Deleting a sandbox on Kubernetes no longer waits out the pod's 30-second termination grace period.

Testing

• On Linux (aarch64, Ubuntu 24.04): cargo clippy -p openshell-sandbox --all-targets -- -D warnings (also with --features perf-harness), mise run rust:format:check, and cargo test -p openshell-sandbox (290 tests) pass
• Unit test added: fails 5 of 5 runs without the fix ("still running 10.0s after SIGTERM") and passes 20 of 20 with it (about 3.1 s)
• E2E tests added/updated (not applicable)

Manual, on k3s with the Kubernetes driver (30 s grace period, a supervisor pod per sandbox), changing only the sandbox runtime image:

	main	this branch
openshell sandbox delete	32.4 s, 33.0 s, 32.0 s	6.3 s, 3.8 s, 4.1 s
agent container exit	SIGKILL (137) at ~31 s	exits on its own at ~3 s
openshell sandbox stop	3.3 s	4.6 s

Stop was already fast in this setup because the driver stops the supervisor pod first.

Follow-ups:

• On Kubernetes the stuck Wait uses the full 2 s bound every time. Waiting up to the existing 2 s reap timeout for the killed agent before serve returns, or recording the exit outside the cancellable reaper, would make the exit immediate.
• The supervisor pod still sometimes ignores SIGTERM and waits out the grace period (2 of 3 deletes above), the supervisor half of bug(sandbox): on Kubernetes, deleting a sandbox waits out the pods' termination grace period #5025. It no longer delays sandbox delete.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (not applicable)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[FrostGod]

/ok to test 0037efe

[FrostGod]

/ok to test f339b5b

[elezar]

Thanks for the fix. The follow-up for the root cause (the agent Wait never returning when shutdown cancels the reaper, so the boundary always uses the full 2 s bound) is now tracked in #5329. Note that #5291 already merged and expects this fix, so landing this PR restores the Kubernetes lifecycle conformance test.

[github-actions]

Label test:e2e applied for f339b5b. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[elezar]

LGTM. The bounded runtime shutdown fixes the hang until SIGKILL, and the root-cause follow-up is tracked in #5329.