Skip to content

Don't hang worker shutdown when a Nexus task is cancelled while encoding its failure - #2020

Merged
tconley1428 merged 2 commits into
temporalio:mainfrom
parbhatkapila4:fix/nexus-cancel-during-encode
Oct 26, 2026
Merged

tconley1428 merged 2 commits into
temporalio:mainfrom
parbhatkapila4:fix/nexus-cancel-during-encode

Conversation

@parbhatkapila4

Copy link
Copy Markdown
Contributor

When a Nexus start or cancel handler fails, the failure is encoded inside the except BaseException block. If the task gets cancelled during that encoding, for example, when the worker shuts down, the CancelledError escapes the block. The outer except Exception doesn't catch it, so _complete_task() is never called and core never gets a completion for the task. Worker shutdown then waits for it forever, which is what the comment on _complete_task() warns about.

Now a cancel during that encode sends an ack_cancel completion, the same thing a cancel during the success path's encode already does.

Added a test with a payload codec that is slow to encode, so worker shutdown cancels the task while it's encoding the failure. It covers the start handler and the cancel handler. On main, shutdown doesn't finish within 10 seconds in either case. With the fix, both pass. The rest of tests/nexus gives the same results with and without this change.

@parbhatkapila4
parbhatkapila4 requested a review from a team as a code owner October 25, 2026 07:12
Comment thread tests/nexus/test_nexus_worker_shutdown.py Outdated

@tconley1428 tconley1428 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Raw sleep should be removed before completion.

@tconley1428
tconley1428 merged commit 01af1f6 into temporalio:main Oct 26, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants