[jrasell]

The plan apply worker can return an refreshed state if it believes the applier process should be retried on a newer state snapshot. This can occur because the quota is blocking the placement or if the follower submitting the plan has an FSM that is lagging behind the rest of the quorum. In this situation the plan apply might be retried, or it might have exhausted its retries and be rejected.

If retries are exhausted, the scheduler will create a blocked eval and set the status details of the evaluation which includes the generated deployment ID. This deployment ID is then persisted to state even though no deployment has been created or itself persisted to state. If the plan and apply was driven by the CLI, this can result in the CLI attempting to monitor a deployment that does not exist.

This change clears the in-memory scheduler deployment object when a state refresh triggers a plan applier retry. If the apply can be retried, this information will be repopulated. If the process has reached the maximum number of retries, this change means the eval no longer references a non-existent deployment.

Testing & Reproduction steps

It's tricky to test manually using CE only code and requires artificially slowing down a follower and ensuring this follower is submitting plans to the leader that will be rejected due to stale state. The linked enterprise PR can be used as a reference on how to manually test this using quota rejections. The unit test covers the scenario quite well.

Links

found while testing: https://github.com/hashicorp/nomad-enterprise/pull/4259

Contributor Checklist

• Changelog Entry If this PR changes user-facing behavior, please generate and add a
  changelog entry using the make cl command.
• Testing Please add tests to cover any new functionality or to demonstrate bug fixes and
  ensure regressions will be caught.
• Documentation If the change impacts user-facing functionality such as the CLI, API, UI,
  and job configuration, please update the Nomad product documentation, which is stored in the
  web-unified-docs repo. Refer to the web-unified-docs contributor guide for docs guidelines.
  Please also consider whether the change requires notes within the upgrade
  guide. If you would like help with the docs, tag the nomad-docs team in this PR.
• LLM Usage If an LLM was used to generate any code, please ensure and confirm you have read
  and followed the AI usage guide.

Reviewer Checklist

• Backport Labels Please add the correct backport labels as described by the internal
  backporting document.
• Commit Type Ensure the correct merge method is selected which should be "squash and merge"
  in the majority of situations. The main exceptions are long-lived feature branches or merges where
  history should be preserved.
• Enterprise PRs If this is an enterprise only PR, please add any required changelog entry
  within the public repository.

[tgross]

This looks right, but there's something related about the if !fullCommit check below that seems odd to me. We check if newState != nil and return early, which implies that by the time we get to result.FullCommit that it is always nil, and then if we didn't get a full comit, we'll always take the error path. These two blocks and the comments around them don't seem to line up.

[tgross]

LOL, and then I opened your other PR #28309 and see you addressed that there.

[tgross]

LGTM!