bug: long-running processes lose provider access after credential refresh #2777

Agent Diagnostic

• Skills used: openshell-cli for the end-to-end sandbox/provider investigation and create-github-issue for this report.
• OpenShell version tested: locally built development binaries reporting openshell 0.0.103-dev.6+gc825b1f8; the source-level proof of concept was validated at checkout 8b7c444c (v0.0.103-17-g8b7c444c) with local uncommitted changes.
• Latest release checked: v0.0.106 (published 2026-08-14). Its release notes do not list a fix for long-running process placeholders. The release artifact was not retested end to end because the reproduction uses a locally built Podman gateway, custom provider refresh configuration, and a local proof-of-concept build.
• Known fixes/designs reviewed: feat(sandbox): supervisor should periodically refresh provider credentials for running sandboxes #1264, feat(providers): add gateway-owned credential refresh for short-lived provider tokens #1306, PR feat(providers): support sandbox provider attach lifecycle #1242, and PR feat(proxy): bind static credentials to provider endpoints #2510. feat(providers): add gateway-owned credential refresh for short-lived provider tokens #1306 owns gateway-side token refresh; its long-running process report describes the same remaining failure after refresh succeeds.
• Possible duplicates reviewed: Enhanced Provider Management #896, Support Codex OAuth in OpenShell using provider-backed auth.json placeholders #988, Provider-managed OAuth credential refresh #1066, feat: support provider attach and detach for running sandboxes #1171, feat(sandbox): supervisor should periodically refresh provider credentials for running sandboxes #1264, and feat(providers): add gateway-owned credential refresh for short-lived provider tokens #1306. No dedicated issue was found for the expired retained placeholder generation or for an identity-stable, revocable workload handle.
• Findings: the gateway refreshed the OAuth access token successfully; a fresh sandbox exec inherited the new provider-environment placeholder and completed a metadata-only request; the existing long-running process retained the old placeholder and failed locally with credential_unavailable after its access token expired. The request did not reach Google. Restarting only the long-running process fixed the request.
• Code diagnosis: provider snapshots are revision-scoped and retain eight generations. Exact lookup of a retained old placeholder wins before same-identity fallback, then fails closed when that generation's token is expired. Routine refresh therefore changes the handle seen by future processes but cannot change the environment inherited by an existing process.
• Proof of concept: keeping an endpoint-bound handle stable across same-identity token updates passed focused tests for an expired initial token, 12 refreshes, non-monotonic revisions, endpoint rebinding, per-key identity replacement, detach/recovery, and replacement revocation. The focused suite passed 35 tests; the full openshell-core suite passed 392 tests with 6 ignored; clippy and relevant crate checks passed.
• Remaining reason for filing: the POC demonstrates the resolver behavior but is not production-ready. The current gateway identity is only <provider object id>:<credential key>, and the POC chooses its stable handle in supervisor memory. Production needs a complete identity-epoch contract and a durable or authenticated handle that survives supervisor/sidecar restart without weakening revocation or endpoint binding.

Description

Actual behavior: A long-running process can lose provider access after OpenShell successfully refreshes a short-lived access token. The process retains a placeholder such as openshell:resolve:env:v1_GOG_ACCESS_TOKEN. A later refresh installs revision v2 for new process launches, but the existing process and its children continue using v1. Once the access token stored in retained generation v1 expires, exact placeholder resolution returns unavailable even though the current same-identity generation contains a valid replacement token.

This affects any process that captures provider environment once and lives across token rotations: agent gateways, editor servers, open PTYs, terminal multiplexers, language servers, and background workers. The application may surface the local proxy failure as an HTTP 500, making it look like an upstream provider outage.

Expected behavior: Routine same-identity access-token refresh should update the credential value behind an identity-bound workload handle. Existing authorized processes should continue working without receiving raw tokens, refresh material, or environment mutation. The handle must fail closed on expiry when refresh fails and must be revoked when its authorization identity or boundary changes.

Root cause

The current model couples two lifetimes that need different semantics:

ProviderCredentialState retains at most eight revision-scoped credential generations. Exact placeholder lookup takes precedence over fallback. If the exact generation still exists but its secret has expired, resolution fails. Same-identity fallback can occur only after an old generation is evicted, so a long-running process can fail for several refresh cycles and later recover as an accidental consequence of retention.

Proposed remediation

Issue one opaque, revocable handle per sandbox/provider credential identity epoch and keep it stable across routine access-token refreshes. Refresh should atomically replace the current access token and expiry behind that handle. Provider environment revision can continue changing for polling, cache invalidation, non-secret environment changes, and future launches; it should not define the lifetime of an already-issued workload authorization handle.

The identity epoch should bind to the equivalent of:

• sandbox identity;
• provider instance and credential key;
• verified OAuth subject/account/tenant identity where available;
• OAuth client and scope or authorization-boundary digest;
• endpoint-binding identity;
• an explicit revocable epoch identifier.

The gateway should issue a durable opaque handle or enough authenticated identity-epoch metadata to reconstruct it consistently. Choosing the stable revision only in supervisor memory is insufficient because a supervisor or sidecar may restart while the workload survives.

Routine same-identity access-token rotation updates the value behind the existing handle. Detach, provider deletion/replacement, account or tenant change, material client/scope change, endpoint-binding identity change, explicit revocation, or sandbox deletion rotates or revokes the epoch. A refresh failure may preserve the current access token only until its recorded expiry, then resolution must fail closed.

This design must preserve the security invariant established by PR #2510: a stale process holding a handle for one provider identity must not resolve a replacement provider's credential merely because both use the same environment key.

Alternatives considered

• Allow fallback from an expired exact generation to the current token only when identity matches. This is a smaller change, but the current identity (provider object id plus credential key) does not include verified subject, tenant, client, scopes, or a binding digest, so reauthorization can be misclassified as continuity.
• Launch every provider client through a fresh supervisor exec. This works around stale environments but requires application-specific brokering and does not fix editors, shells, or ordinary child-process inheritance.
• Restart long-running processes after every refresh. This is the current workaround and interrupts sessions, sockets, editor state, and in-flight work roughly once per access-token lifetime.
• Keep old access tokens usable indefinitely. Rejected: expiry must remain fail-closed, and refresh material must never enter the workload.

Reproduction Steps

1. Configure a provider with gateway-owned OAuth refresh material and a short-lived injectable access-token credential.
2. Attach it to a running sandbox and start a long-running parent process from a supervisor-created exec.
3. Confirm, without printing credential values, that the process inherited revision-scoped placeholder P1.
4. From that process, run a metadata-only provider operation and confirm success.
5. Wait for a successful gateway refresh and provider-environment revision change.
6. Confirm that a fresh supervisor exec inherits placeholder P2 and succeeds with the same metadata-only operation.
7. After the token associated with P1 expires, launch the provider client from the original parent process.
8. Observe local credential_unavailable/expired-placeholder diagnostics and confirm the request does not reach the provider.
9. Restart only the long-running parent through a fresh supervisor exec and observe that the operation succeeds.

For Google Workspace with gog, a content-free probe is:

gog --readonly --no-input --json gmail labels list >/dev/null

Do not print environment values or token-bearing request data while reproducing.

Environment

• OS: Fedora Linux 44 (Workstation Edition)
• Sandbox backend: local Podman
• OpenShell end-to-end binary: 0.0.103-dev.6+gc825b1f8
• POC source checkout: 8b7c444c (v0.0.103-17-g8b7c444c) with local uncommitted POC changes
• Workload: OpenClaw launching gog for a metadata-only Google Workspace Gmail request
• Credential lifecycle: gateway-owned OAuth refresh token; short-lived access token exposed to the sandbox only through an OpenShell placeholder
• Latest release checked: v0.0.106; release notes reviewed, no matching fix listed; release artifact not retested for the reason given in Agent Diagnostic
• Possible duplicates checked: yes; Enhanced Provider Management #896, Support Codex OAuth in OpenShell using provider-backed auth.json placeholders #988, Provider-managed OAuth credential refresh #1066, feat: support provider attach and detach for running sandboxes #1171, feat(sandbox): supervisor should periodically refresh provider credentials for running sandboxes #1264, and feat(providers): add gateway-owned credential refresh for short-lived provider tokens #1306

Logs

# Redacted OpenShell diagnostics; no credential values or mailbox data.
credential resolution rejected: credential is expired location=resolve_placeholder
reason=credential_unavailable

# Control observations:
gateway refresh status: successful
fresh supervisor exec using current placeholder: request successful
existing long-running process using retained placeholder: request failed locally
provider request observed: no
restart of long-running process only: request successful

Acceptance Criteria

• A process started before refresh continues making authorized requests through at least 12 consecutive same-identity access-token rotations.
• Access tokens and refresh material never enter the process environment; refresh tokens, client secrets, and service-account private material remain non-injectable gateway state.
• Handles survive supervisor/sidecar restart when the workload survives.
• Detach, deletion, explicit revocation, or provider/account/subject/tenant/authorization replacement invalidates old handles.
• An old handle cannot cross provider identity, OAuth subject, client/scope authorization boundary, or endpoint binding.
• Failed refresh preserves the current token only until recorded expiry, then fails closed without a failure window tied to generation eviction.
• Endpoint and L7 policy checks still run before substitution; endpoint-bound credentials cannot be replayed to another host, port, or path.
• Docker, Podman, Kubernetes sidecar, and VM supervisor topologies have equivalent behavior.
• E2E coverage uses a fake short-lived token issuer and includes long-running-process, refresh, revocation, replacement, refresh failure, and supervisor/sidecar restart cases.
• OCSF/application diagnostics distinguish local credential unavailability from an upstream HTTP 500 without logging placeholders, tokens, keys, query data, or user content.

Related Work

• feat(providers): add gateway-owned credential refresh for short-lived provider tokens #1306 owns gateway-side refresh material and token rotation.
• Matching long-running-process report on #1306
• feat(sandbox): supervisor should periodically refresh provider credentials for running sandboxes #1264 proposed stable request-time resolver updates.
• PR feat(providers): support sandbox provider attach lifecycle #1242 introduced provider attach/detach generation snapshots.
• PR feat(proxy): bind static credentials to provider endpoints #2510 added provider identity and endpoint binding that this fix must preserve.

Agent-First Checklist

• I pointed my agent at the repo and had it investigate this issue.
• I loaded relevant skills.
• I checked the latest OpenShell release and explained why the artifact was not retested.
• I searched existing issues for possible duplicates.
• The diagnostic explains why the issue remains unresolved.