[tgross]

An evaluation that's in-flight in the scheduler when the job is garbage collected, will fail and eventually reach the delivery limit. When the leader is pulling failed evals off the eval broker, it creates a new follow-up eval which will meet the same fate and this will continue forever (albeit slowly due to backoff) until the evals are manually deleted.

Update the failed evals reaper to check the state for the evaluation being "orphaned": no job in the state store. It appears that we can't check this unconditionally on all eval upserts in the FSM because there are many cases where we want to update evals where the job has already been deleted.

It appears that we can't check this unconditionally on upsert in the FSM because there are many cases where we want to update evals where the job has already been deleted.

Ref: https://hashicorp.atlassian.net/browse/NMD-1012
Fixes: #16877

Contributor Checklist

• Changelog Entry If this PR changes user-facing behavior, please generate and add a
  changelog entry using the make cl command.
• Testing Please add tests to cover any new functionality or to demonstrate bug fixes and
  ensure regressions will be caught.
• Documentation If the change impacts user-facing functionality such as the CLI, API, UI,
  and job configuration, please update the Nomad product documentation, which is stored in the
  web-unified-docs repo. Refer to the web-unified-docs contributor guide for docs guidelines.
  Please also consider whether the change requires notes within the upgrade
  guide. If you would like help with the docs, tag the nomad-docs team in this PR.

Reviewer Checklist

• Backport Labels Please add the correct backport labels as described by the internal
  backporting document.
• Commit Type Ensure the correct merge method is selected which should be "squash and merge"
  in the majority of situations. The main exceptions are long-lived feature branches or merges where
  history should be preserved.
• Enterprise PRs If this is an enterprise only PR, please add any required changelog entry
  within the public repository.

• If a change needs to be reverted, we will roll out an update to the code within 7 days.

Changes to Security Controls

Are there any changes to security controls (access controls, encryption, logging) in this pull request? If so, explain.

[mismithhisler]

Looks good!

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.