Provider plugin processes keep running after Terraform panics (os.Exit(11) in PanicHandler skips plugin.CleanupClients) #39399

Terraform Version

Terraform v1.16.6
on linux_amd64
+ provider registry.terraform.io/hashicorp/random v3.9.1

Also reproduced with v1.15.8, v1.15.9, and a build of main at 9ec49e0 (v1.18.0-dev).

Terraform Configuration Files

main.tf:

terraform {
  required_providers {
    random = {
      source = "hashicorp/random"
    }
  }
}

resource "random_password" "this" {
  count  = 1
  length = 16
}

tests/main.tftest.hcl:

mock_provider "random" {}

run "assertion_fails" {
  command = apply

  assert {
    condition     = length(random_password.this) == 0
    error_message = "expected no password"
  }
}

The failing assertion triggers the panic tracked in #39365. That panic is used here only because it is a deterministic way to make Terraform crash while a provider plugin is running.

Debug Output

A full TF_LOG=trace log is available on request. Panic excerpt:

panic: unexpected error marshalling value: value has marks, so it cannot be serialized as JSON
goroutine 86 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:26 +0x5e
github.com/hashicorp/terraform/internal/logging.PanicHandler()
	github.com/hashicorp/terraform/internal/logging/panic.go:84 +0x15c
...
github.com/hashicorp/terraform/internal/command/views/json.NewDiagnostic({0x4ab02a0, 0x35a38c928120}, 0x35a38ca54c30)
	github.com/hashicorp/terraform/internal/command/views/json/diagnostic.go:278 +0x18de
...
github.com/hashicorp/terraform/internal/moduletest/graph.(*NodeTestRun).Execute(0x35a38c4c2cf0, 0x35a38bfc1260)
	github.com/hashicorp/terraform/internal/moduletest/graph/node_test_run.go:147 +0x4b8
github.com/hashicorp/terraform/internal/moduletest/graph.Walk.func1({0x4445fa0, 0x35a38c4c2cf0})
	github.com/hashicorp/terraform/internal/moduletest/graph/test_graph_builder.go:130 +0x257

In the trace log, the random plugin is started (provider: plugin started: ... pid=...) and never logs plugin process exited.

Expected Behavior

When Terraform crashes, it still terminates the provider plugin processes it started, as it does on a normal exit. No terraform-provider-* process outlives the terraform process.

Actual Behavior

terraform test exits with status 11 after the crash banner. The terraform-provider-random process it started keeps running. It is reparented to the init or subreaper process and is still running two minutes later:

$ terraform test; echo "exit=$?"
...
exit=11
$ ps -o pid,ppid,stat,etime,rss,comm -p 3250962   # 10 s after exit
    PID    PPID STAT     ELAPSED   RSS COMMAND
3250962     594 Sl         00:10 21504 terraform-provi
$ ps -o pid,ppid,stat,etime,rss,comm -p 3250962   # 120 s after exit
    PID    PPID STAT     ELAPSED   RSS COMMAND
3250962     594 Sl         02:00 10496 terraform-provi

PID 594 is the session's subreaper. The process has to be killed by hand. With a large provider such as hashicorp/aws, each orphan holds roughly 375-380 MB RSS. Repeated crashes in CI or in scripted test loops accumulate these processes.

Steps to Reproduce

1. Save the two files above.
2. terraform init
3. terraform test; echo "exit=$?" prints exit=11 after the crash banner.
4. pgrep -a terraform-provider lists the random plugin that is still running.

No credentials are needed.

Additional Context

The plugin is orphaned only when a provider instance is still alive at the moment of the panic. In terraform test, that is the case when the test file declares a provider or mock_provider block:

TestProvidersTransformer (internal/moduletest/graph/transform_providers.go) starts file-level providers while it builds the graph. They are closed only by NodeProviderClose, after all runs. Per-run providers are closed before the run's result is rendered, so they are already gone when the panic happens.

The orphaning itself is not specific to terraform test:

• Worker goroutines defer logging.PanicHandler(). That function ends with os.Exit(11) (internal/logging/panic.go:89).
• os.Exit skips deferred calls, including defer plugin.CleanupClients() in realMain (main.go:248).
• Managed plugins (internal/command/meta_providers.go:467) are therefore never killed.

The same defer logging.PanicHandler() pattern guards goroutines in internal/terraform/graph.go, internal/terraform/context.go, internal/backend/local/backend_plan.go, backend_apply.go and backend_refresh.go. A panic in any of them would orphan running plugins. Only the terraform test path is demonstrated here.

A local experiment that calls plugin.CleanupClients() in PanicHandler immediately before os.Exit(11) removed the orphans in every case tried. Those cases were the configuration above, the same with provider "random" {}, plan mode, and a module using hashicorp/aws under mock_provider. The exit status stayed 11 and the crash report was unchanged. Whether that is the right place for the cleanup is open for discussion.

References

• terraform test panics ("value has marks") when a failed assertion references a whole object that has deprecated attributes #39365: the terraform test panic used as the trigger here.
• terraform test crashes when an assertion fails on a deprecation-marked value #39239: the same panic. It notes that the crash also skips the test's apply cleanup.

Generative AI / LLM assisted development?

An LLM coding agent helped narrow down the reproduction and read the source. Every result above was run and checked on the listed versions.