Repository navigation
Conversation
During shutdown, several components log expected shutdown-ordering races at WARN/ERROR with full stack traces, drowning out real problems: - ScheduledExecutors.scheduleAtFixedRate logged a WARN with a full stack trace for any exception from a scheduled task, even when the executor was already shut down and the task would not be rescheduled anyway. Now checks exec.isShutdown() in the catch block and logs a single DEBUG line without a stack trace in that case. - MetadataSegmentView.stop() used scheduledExec.shutdown(), which does not interrupt an in-flight poll. If the Coordinator client's DiscoveryServiceLocator has already stopped by the time that poll completes, it fails with ServiceClosedException, logged via the above. Switched to shutdownNow() so a blocked poll is interrupted immediately instead. - ChangeRequestHttpSyncer's HTTP callback was submitted directly to its ScheduledExecutorService, so a callback arriving after HttpServerInventoryView.stop() has torn down that executor causes a RejectedExecutionException, which Guava's AbstractFuture logs at SEVERE once per in-flight connection (hundreds on a large cluster). The callback is now submitted through a small wrapper that catches RejectedExecutionException and logs at DEBUG instead; dropping the callback is safe since it already no-ops via startStopLock.awaitStarted(...) when the syncer is not started. - HttpServerInventoryView.stop() now stops each DruidServerHolder (and its ChangeRequestHttpSyncer) before shutting down the shared sync/monitoring executors, so no new sync requests are issued during the shutdown window. A failure to stop one server's holder is logged and does not prevent the rest from stopping or the executors from shutting down. None of these changes affect normal operation; they only change what gets logged, and how promptly resources are released, during shutdown.
FrankChen021
left a comment
There was a problem hiding this comment.
🟡 Changes recommended
The shutdown-aware logging is incomplete: the new regression test and MetadataSegmentView use scheduleAtFixedRate, but that helper still logs shutdown rejections at WARN, and discovery callbacks can race with the shutdown snapshot and start holders after shutdown begins.
Reviewed 8 of 8 changed files; findings in 2 files.
Validation: git diff --check on the prepared merge-base-to-head diff reported three whitespace diagnostics. No build or tests were run.
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 1 |
| P2 | 1 |
| P3 | 0 |
| Total | 2 |
This is an automated review by Codex GPT-5.6-Luna(max)
After addressing the findings or replying to the comments, you can request another review from me to trigger a new automated review.
|
the PR description is also too verbose. you should ask agent (if you're using) to generate concise one. |
Fix P1: the shutdown-aware logging check had been applied to scheduleWithFixedDelay instead of scheduleAtFixedRate, which MetadataSegmentView and the regression test actually use. Moved it to the correct method and reverted scheduleWithFixedDelay to its original behavior. Fix P2: HttpServerInventoryView.stop() now synchronizes with serverAdded() via a shuttingDown flag guarded by the servers monitor, so a discovery callback racing with shutdown cannot start a new DruidServerHolder after the shutdown snapshot is taken. Also replaced a deprecated new URL(...) call in the new regression test with URI.create(...).toURL().
|
Hi @FrankChen021, I took a note of your suggested corrections and have applied them. Could you please review the changes and lmk if anything needs further improvements? |
FrankChen021
left a comment
There was a problem hiding this comment.
🟡 Changes recommended
Follow-up assessment
Thread 4550509072 is fixed: the shutdown-aware logging branch now guards scheduleAtFixedRate, while scheduleWithFixedDelay retains its original behavior.
Thread 4550509077's original add-versus-stop race is fixed by the synchronized shuttingDown flag and holder snapshot. However, the stopped snapshot entries remain in servers; a late discovery-removal or monitoring-reset callback can call serverRemoved() and stop the same holder a second time, causing shutdown errors.
Reviewed 8 of 8 changed files; finding in 1 file. This was a static follow-up; no runtime validation is claimed.
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 0 |
| P2 | 1 |
| P3 | 0 |
| Total | 1 |
This is an automated review by Codex GPT-5.6-Luna(max)
After addressing the findings or replying to the comments, you can request another review from me to trigger a new automated review.
stop() now clears the servers map while holding the servers monitor, in addition to snapshotting it, so a late nodesRemoved or checkAndResetUnhealthyServers callback finds nothing to remove and never reaches serverRemoved()/DruidServerHolder.stop() for an already-stopped holder. DruidServerHolder.stop() is also now idempotent via AtomicBoolean.compareAndSet, as a defensive backstop against any other path that might call it twice.
|
Done with the suggested changes, would appreciate another review and your opinions on what can be improved. TIA! |
FrankChen021
left a comment
There was a problem hiding this comment.
🟡 Changes recommended
Fix the metadata cache initialization wait before merging. The reported HttpServerInventoryView double-stop race is fixed, but the new forced interruption can strand a sys.segments caller indefinitely.
Follow-up assessment
The reported P2 is fixed at the current head: HttpServerInventoryView.stop() clears servers under its monitor before stopping the snapshot, and DruidServerHolder.stop() is idempotent. One remaining P1 is in MetadataSegmentView.stop(): interrupting the initial poll can prevent cachePopulated.countDown(), leaving getSegments() blocked forever.
Reviewed 8 of 8 changed files; finding in 1 file. This was a static current-head follow-up; no tests or builds were run.
Validation: git diff --check passed.
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 1 |
| P2 | 0 |
| P3 | 0 |
| Total | 1 |
This is an automated review by Codex GPT-5.6-Luna(max)
After addressing the findings or replying to the comments, you can request another review from me to trigger a new automated review.
MetadataSegmentView.stop() now also counts down cachePopulated after shutdownNow(), so a getSegments() caller blocked on the initial poll is released rather than stuck forever if that poll is interrupted before it ever reaches cachePopulated.countDown(). getSegments() checks a new stopped flag immediately after the cache-populated wait and throws a clear ISE instead of reading the now-nulled cache. Added a regression test (test_stop_releasesWaitersBlockedOnInitialPoll) that blocks the first poll, calls stop() concurrently, and asserts the waiting getSegments() call unblocks with that exception rather than hanging.
|
hi @FrankChen021, could I get another review now? |
FrankChen021
left a comment
There was a problem hiding this comment.
🟢 Approval recommended
The previously reported MetadataSegmentView initialization deadlock is resolved at the reviewed head: shutdown marks the view stopped, releases cachePopulated after interrupting the poll, and waiters fail clearly instead of remaining blocked. The surrounding shutdown changes were rechecked and no remaining actionable issue was found.
Follow-up assessment
The original P1 is fixed, and the added regression test covers a waiter blocked on the initial poll. This was a static current-head follow-up.
Reviewed 8 of 8 changed files.
This is an automated review by Codex GPT-5.6-Luna(max)
Fixes #20873.
Description
When a Broker (or other service) shuts down, several components log noisy
WARN/ERROR stack traces for expected shutdown-ordering races, drowning out
real problems. This PR quiets two sources of that noise.
ScheduledExecutors.scheduleAtFixedRateThe catch block logged every exception at WARN with a full stack trace, even
when the executor was already shut down and the task would never be
rescheduled anyway (e.g.
MetadataSegmentView's poll failing withServiceClosedExceptiononce the Coordinator's locator has also stopped).It now checks
exec.isShutdown()first and logs at DEBUG without a stacktrace in that case.
MetadataSegmentView.stop()also now usesshutdownNow()instead ofshutdown(), so a blocked poll is interruptedimmediately rather than potentially failing after the fact.
ChangeRequestHttpSyncer/HttpServerInventoryViewA
ChangeRequestHttpSyncercallback arriving after its executor hasterminated caused Guava to log a
RejectedExecutionExceptionat SEVERE —once per open connection, which can mean hundreds of stack traces on a large
cluster. The callback is now submitted through a wrapper that catches this
exception and logs it at DEBUG; dropping the callback is safe since it
already no-ops when the syncer is not started.
HttpServerInventoryView.stop()now stops each tracked server's syncerbefore shutting down the shared executors, and rejects any server added
after shutdown begins (synchronized on the same lock as
serverAdded), sono new sync work can start during the shutdown window.
Release note
Services no longer log noisy WARN/ERROR stack traces for expected
shutdown-ordering races when a Broker or other Druid service shuts down.
Key changed/added classes in this PR
ScheduledExecutorsMetadataSegmentViewChangeRequestHttpSyncerHttpServerInventoryViewThis PR has: