Skip to content

fix: complete tasks killed while HRTR waits for the worker to report them - #20890

Open
amaechler wants to merge 3 commits into
apache:masterfrom
amaechler:hrtr-kill-during-assignment-wait
Open

amaechler wants to merge 3 commits into
apache:masterfrom
amaechler:hrtr-kill-during-assignment-wait

Conversation

@amaechler

Copy link
Copy Markdown
Contributor

Follow-up to #20773 and #20867.

Description

After the Overlord sends a task to a MiddleManager or Indexer, HttpRemoteTaskRunner waits up to druid.indexer.runner.taskAssignmentTimeout (5 minutes by default) for the worker to report it. It also waits when the assign request fails and the Overlord cannot tell whether the task arrived. The runner ignored a kill during this wait until the timeout: the task kept counting as pending, so autoscaling could treat it as demand, and the timeout then raised a "Task assignment timed out" alert for a task that was already gone.

Fix

shutdown() now wakes the waiting thread, which marks the killed task FAILED right away. The worker stays reserved until it reports the task or the timeout passes, because it may hold the task without the Overlord knowing, and workers accept tasks beyond their capacity. If the worker reports the task, the Overlord tells it to stop, as it already does for completed tasks. A killed task no longer raises the timeout alert.

Tests

The new testShutdownTaskWhileWaitingForWorkerToReportIt covers a worker that never reports the killed task, reports it RUNNING, or reports it FAILED. All three cases fail without the fix. testShutdownTaskBeingAssigned(true) drops two assertions that depended on whether the kill or the worker's report arrived first.

Release note

A task killed while the Overlord waits for a worker to confirm its assignment no longer stays pending for up to 5 minutes or raises a "Task assignment timed out" alert.


This PR has:

  • been self-reviewed.
  • a release note entry in the PR description.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.

When WorkerHolder.assignTask() returns true, including its "not sure
whether assigned" path after every retry fails, runTaskOnWorker() waits
up to taskAssignmentTimeout for the worker to report the task. A kill
during that wait was not acted on: the task stayed in getPendingTasks()
and getPendingTaskPayloads() with its future open, then failed with the
timeout reason and raised the "Task assignment timed out" alert.

shutdown() now wakes the assigning thread after recording the kill. The
thread completes the task with the kill reason, then keeps the worker
reserved until the worker reports the task (WorkerHolder.hasAnnouncement)
or the original deadline passes, because workers do not reject
assignments beyond their capacity. If the worker reports the task
RUNNING, taskAddedOrUpdated() asks it to shut the task down.
It mostly re-tested the shutdown on a RUNNING report from apache#20867, which testShutdownSentAgainWhenKilledTaskIsReportedRunning and testShutdownTaskBeingAssigned already cover.

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Fix the worker-side completion and acknowledgement handling before merging. The early Overlord completion currently bypasses worker completion bookkeeping, and a worker update that is removed in the same sync batch can leave the worker reserved until the assignment timeout.

Reviewed 3 of 3 changed files; findings in 2 production files. Coverage included the full merge-base diff, surrounding assignment/shutdown state transitions, worker announcement snapshot and delta-sync behavior, task-runner call sites, and the added parameterized tests.

Validation: git diff --check c951187c8bd59718b9dc3590bf1522e80567ec11..HEAD passed. Build and tests were not run; this was a static review.

Severity Findings
P0 0
P1 0
P2 2
P3 0
Total 2

This is an automated review by Codex GPT-5.6 Luna(Max)

After addressing the findings or replying to the comments, you can request another review from me to trigger a new automated review.

if (isTaskAssignmentTimedOut) {
if (shutdownStatus != null) {
// taskComplete(..) must be called outside of statusLock, see comments on method.
taskComplete(workItem, null, shutdownStatus);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve worker completion bookkeeping after early kill completion

Finding: After a killed assignment wakes this branch, taskComplete marks the work item COMPLETE before the worker has acknowledged it. If the worker reports RUNNING and is then shut down, or reports FAILED/SUCCESS, taskAddedOrUpdated sees the already-COMPLETE item and takes its COMPLETE branch, so it never calls taskComplete with workerHolder. That skips setLastCompletedTaskTime and blacklistWorkerIfNeeded, leaving idle-worker and consecutive-failure bookkeeping stale for a task that actually ran on this worker.

Suggestion: Keep the worker-side terminal announcement on the normal completion path, or explicitly perform the WorkerHolder completion bookkeeping when a post-kill announcement arrives without changing the Overlord task's already-recorded kill status.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not counting the kill against the worker is intended: the task failed because it was killed, not because of the worker, so it should not move the worker toward blacklisting. lastCompletedTaskTime only feeds the idle-worker check, and the killed task ran for at most the time it took the shutdown to land. Doing this bookkeeping for late reports in general would need to tell a first terminal report from a re-report, since workers re-send completed tasks on every full sync.

// The worker may have accepted the task, so keep it reserved until it reports the task or the timeout passes.
// If it reports the task running, taskAddedOrUpdated() asks it to kill the task.
synchronized (statusLock) {
while (!workerHolder.hasAnnouncement(taskId)) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Do not miss acknowledgements removed in the same worker sync

Finding: hasAnnouncement only inspects the final tasksSnapshotRef. WorkerHolder applies an entire delta batch to that snapshot before notifying HRTR, so an accepted task can produce a TaskUpdate followed by TaskRemoval in one batch and leave no entry even though taskAddedOrUpdated already received the report. The waiter then misses the acknowledgement and retains workersWithUnacknowledgedTask until the full assignment timeout, blocking subsequent assignments to that worker.

Suggestion: Record or propagate the observed TaskAnnouncement, including terminal updates before removal, to the waiter, or release the reservation when taskAddedOrUpdated handles the assigned task instead of relying only on the final snapshot.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in fe1f891. taskAddedOrUpdated() now releases the reservation when the reserved worker reports the task, and the wait checks the reservation instead of the announcement snapshot, so an update that is removed in the same sync batch is no longer missed. The test now delivers the FAILED update and the removal in one deltaSync batch.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed fixed in fe1f891. I reviewed 2 of 2 changed files; the waiter now observes the worker/task reservation, so a task update followed by removal in one delta batch no longer leaves the worker reserved until timeout.

The wait for a killed task checked the worker's announcement snapshot, so it missed a report that the worker removed in the same sync batch and held the worker until the assignment timeout. taskAddedOrUpdated() now releases the reservation when the reserved worker reports the task, and the assigning thread only removes its own reservation.

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

Follow-up assessment

Both prior findings were rechecked at the current head. The acknowledgement race is fixed, and the worker bookkeeping concern is intentional rather than an actionable defect.

Reviewed 2 of 2 changed files. Static review only; no tests or builds were run.


This is an automated review by Codex GPT-5.5

@FrankChen021 FrankChen021 added the Approval Recommended Used when AI completes a review and reports no findings label Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Approval Recommended Used when AI completes a review and reports no findings Area - Ingestion Bug Release Notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants