Skip to content

fix: suppress noisy stack traces during broker shutdown - #20882

Open
tyrobro wants to merge 5 commits into
apache:masterfrom
tyrobro:fix/quiet-shutdown-logging
Open

tyrobro wants to merge 5 commits into
apache:masterfrom
tyrobro:fix/quiet-shutdown-logging

Conversation

@tyrobro

@tyrobro tyrobro commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #20873.

Description

When a Broker (or other service) shuts down, several components log noisy
WARN/ERROR stack traces for expected shutdown-ordering races, drowning out
real problems. This PR quiets two sources of that noise.

ScheduledExecutors.scheduleAtFixedRate

The catch block logged every exception at WARN with a full stack trace, even
when the executor was already shut down and the task would never be
rescheduled anyway (e.g. MetadataSegmentView's poll failing with
ServiceClosedException once the Coordinator's locator has also stopped).
It now checks exec.isShutdown() first and logs at DEBUG without a stack
trace in that case. MetadataSegmentView.stop() also now uses
shutdownNow() instead of shutdown(), so a blocked poll is interrupted
immediately rather than potentially failing after the fact.

ChangeRequestHttpSyncer / HttpServerInventoryView

A ChangeRequestHttpSyncer callback arriving after its executor has
terminated caused Guava to log a RejectedExecutionException at SEVERE —
once per open connection, which can mean hundreds of stack traces on a large
cluster. The callback is now submitted through a wrapper that catches this
exception and logs it at DEBUG; dropping the callback is safe since it
already no-ops when the syncer is not started.

HttpServerInventoryView.stop() now stops each tracked server's syncer
before shutting down the shared executors, and rejects any server added
after shutdown begins (synchronized on the same lock as serverAdded), so
no new sync work can start during the shutdown window.

Release note

Services no longer log noisy WARN/ERROR stack traces for expected
shutdown-ordering races when a Broker or other Druid service shuts down.


Key changed/added classes in this PR
  • ScheduledExecutors
  • MetadataSegmentView
  • ChangeRequestHttpSyncer
  • HttpServerInventoryView

This PR has:

  • been self-reviewed, including the concurrency checklist.
  • a release note entry in the PR description.
  • added comments explaining the "why" where not obvious.
  • added or updated unit tests to cover the new code paths.

During shutdown, several components log expected shutdown-ordering
races at WARN/ERROR with full stack traces, drowning out real
problems:

- ScheduledExecutors.scheduleAtFixedRate logged a WARN with a full
  stack trace for any exception from a scheduled task, even when the
  executor was already shut down and the task would not be
  rescheduled anyway. Now checks exec.isShutdown() in the catch block
  and logs a single DEBUG line without a stack trace in that case.

- MetadataSegmentView.stop() used scheduledExec.shutdown(), which
  does not interrupt an in-flight poll. If the Coordinator client's
  DiscoveryServiceLocator has already stopped by the time that poll
  completes, it fails with ServiceClosedException, logged via the
  above. Switched to shutdownNow() so a blocked poll is interrupted
  immediately instead.

- ChangeRequestHttpSyncer's HTTP callback was submitted directly to
  its ScheduledExecutorService, so a callback arriving after
  HttpServerInventoryView.stop() has torn down that executor causes
  a RejectedExecutionException, which Guava's AbstractFuture logs at
  SEVERE once per in-flight connection (hundreds on a large cluster).
  The callback is now submitted through a small wrapper that catches
  RejectedExecutionException and logs at DEBUG instead; dropping the
  callback is safe since it already no-ops via
  startStopLock.awaitStarted(...) when the syncer is not started.

- HttpServerInventoryView.stop() now stops each DruidServerHolder
  (and its ChangeRequestHttpSyncer) before shutting down the shared
  sync/monitoring executors, so no new sync requests are issued
  during the shutdown window. A failure to stop one server's holder
  is logged and does not prevent the rest from stopping or the
  executors from shutting down.

None of these changes affect normal operation; they only change
what gets logged, and how promptly resources are released, during
shutdown.
@tyrobro tyrobro changed the title fix: suppress noisy stack traces during broker shutdown fix: suppress noisy stack traces during broker shutdown Nov 11, 2026

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The shutdown-aware logging is incomplete: the new regression test and MetadataSegmentView use scheduleAtFixedRate, but that helper still logs shutdown rejections at WARN, and discovery callbacks can race with the shutdown snapshot and start holders after shutdown begins.

Reviewed 8 of 8 changed files; findings in 2 files.

Validation: git diff --check on the prepared merge-base-to-head diff reported three whitespace diagnostics. No build or tests were run.

Severity Findings
P0 0
P1 1
P2 1
P3 0
Total 2

This is an automated review by Codex GPT-5.6-Luna(max)

After addressing the findings or replying to the comments, you can request another review from me to trigger a new automated review.

@FrankChen021

Copy link
Copy Markdown
Member

the PR description is also too verbose. you should ask agent (if you're using) to generate concise one.

Fix P1: the shutdown-aware logging check had been applied to scheduleWithFixedDelay instead of scheduleAtFixedRate, which MetadataSegmentView and the regression test actually use. Moved it to the correct method and reverted scheduleWithFixedDelay to its original behavior.

Fix P2: HttpServerInventoryView.stop() now synchronizes with serverAdded() via a shuttingDown flag guarded by the servers monitor, so a discovery callback racing with shutdown cannot start a new DruidServerHolder after the shutdown snapshot is taken.

Also replaced a deprecated new URL(...) call in the new regression test with URI.create(...).toURL().
@tyrobro

tyrobro commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

Hi @FrankChen021, I took a note of your suggested corrections and have applied them. Could you please review the changes and lmk if anything needs further improvements?

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Follow-up assessment

Thread 4550509072 is fixed: the shutdown-aware logging branch now guards scheduleAtFixedRate, while scheduleWithFixedDelay retains its original behavior.

Thread 4550509077's original add-versus-stop race is fixed by the synchronized shuttingDown flag and holder snapshot. However, the stopped snapshot entries remain in servers; a late discovery-removal or monitoring-reset callback can call serverRemoved() and stop the same holder a second time, causing shutdown errors.

Reviewed 8 of 8 changed files; finding in 1 file. This was a static follow-up; no runtime validation is claimed.

Severity Findings
P0 0
P1 0
P2 1
P3 0
Total 1

This is an automated review by Codex GPT-5.6-Luna(max)

After addressing the findings or replying to the comments, you can request another review from me to trigger a new automated review.

stop() now clears the servers map while holding the servers monitor, in addition to snapshotting it, so a late nodesRemoved or checkAndResetUnhealthyServers callback finds nothing to remove and never reaches serverRemoved()/DruidServerHolder.stop() for an already-stopped holder.

DruidServerHolder.stop() is also now idempotent via AtomicBoolean.compareAndSet, as a defensive backstop against any other path that might call it twice.
@tyrobro

tyrobro commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

Done with the suggested changes, would appreciate another review and your opinions on what can be improved. TIA!

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Fix the metadata cache initialization wait before merging. The reported HttpServerInventoryView double-stop race is fixed, but the new forced interruption can strand a sys.segments caller indefinitely.

Follow-up assessment

The reported P2 is fixed at the current head: HttpServerInventoryView.stop() clears servers under its monitor before stopping the snapshot, and DruidServerHolder.stop() is idempotent. One remaining P1 is in MetadataSegmentView.stop(): interrupting the initial poll can prevent cachePopulated.countDown(), leaving getSegments() blocked forever.

Reviewed 8 of 8 changed files; finding in 1 file. This was a static current-head follow-up; no tests or builds were run.

Validation: git diff --check passed.

Severity Findings
P0 0
P1 1
P2 0
P3 0
Total 1

This is an automated review by Codex GPT-5.6-Luna(max)

After addressing the findings or replying to the comments, you can request another review from me to trigger a new automated review.

MetadataSegmentView.stop() now also counts down cachePopulated after shutdownNow(), so a getSegments() caller blocked on the initial poll is released rather than stuck forever if that poll is interrupted before it ever reaches cachePopulated.countDown().

getSegments() checks a new stopped flag immediately after the cache-populated wait and throws a clear ISE instead of reading the now-nulled cache.

Added a regression test (test_stop_releasesWaitersBlockedOnInitialPoll) that blocks the first poll, calls stop() concurrently, and asserts the waiting getSegments() call unblocks with that exception rather than hanging.
@tyrobro

tyrobro commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

hi @FrankChen021, could I get another review now?

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The previously reported MetadataSegmentView initialization deadlock is resolved at the reviewed head: shutdown marks the view stopped, releases cachePopulated after interrupting the poll, and waiters fail clearly instead of remaining blocked. The surrounding shutdown changes were rechecked and no remaining actionable issue was found.

Follow-up assessment

The original P1 is fixed, and the added regression test covers a waiter blocked on the initial poll. This was a static current-head follow-up.

Reviewed 8 of 8 changed files.


This is an automated review by Codex GPT-5.6-Luna(max)

@FrankChen021 FrankChen021 added the Approval Recommended Used when AI completes a review and reports no findings label Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Approval Recommended Used when AI completes a review and reports no findings Area - Querying

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Broker shutdown logs many spurious stack traces from MetadataSegmentView and ChangeRequestHttpSyncer

3 participants