Repository navigation
Description
Search before asking
- I had searched in the issues and found no similar issues.
Version
3.0.x / 4.1.x / master
What's Wrong?
When a client application (e.g., a microservice or web backend with a 3–5 second query timeout) disconnects due to client-side timeout while the query is still waiting on the BE (e.g. waiting for rowset/delete-bitmap lock or waiting in queue during heavy ingestion/compaction), the following cascade failure occurs:
-
FE Network Layer Removes Connection Silently:
FE'sAcceptListenerdetects the TCP FIN/RST from the client and triggersconnection.setCloseListener(...)->connectScheduler.getConnectPoolMgr().unregisterConnection(context). The connection is removed fromconnectionMap. -
No Cancel Signal Is Dispatched:
ConnectPoolMgr.unregisterConnection()removes the context fromconnectionMap, but does NOT callcontext.cancelQuery(). No cancel signal is sent to theCoordinator, and nocancel_plan_fragmentRPC is dispatched to BE nodes. -
Query Escapes
TimeoutCheckerEntirely (Ghost Query):
FE's backgroundTimeoutChecker(checkTimer) iterates strictly overconnectionMap.values(). Because the connection was already removed in step 1,checkTimeout()is never called again for this context. The query bypassesquery_timeout(e.g. 300s) and hangs indefinitely inQeProcessorImpl/information_schema.active_queries(observed running for >3400 seconds / 57 minutes asRUNNING). -
FE Worker Thread Stalls in
coordBase.getNext():
Because the query was actively executing andReadListener.suspendAcceptQuery()had already suspended reading on the socket, the worker thread remains blocked waiting for BE results. Since no data is written to the closed socket, noIOException/EPIPEis raised to break the loop. -
Workload Group Queue Slot Leak & Cluster Stall:
BecauseCoordinator.close()is never executed, the query'sQueueTokenis never returned to theQueryQueue. When all slots (max_concurrency) in the workload group are occupied by these orphaned queries, all subsequent queries in that workload group are stuck inWAIT_IN_QUEUEforever, until they fail withquery queue timeout.
What You Expected?
When a client disconnects or closes the connection:
unregisterConnection()must immediately cancel any active query on that connection.- The
Coordinatormust abort BE fragment execution via cancel RPCs, release the workload group'sQueueToken, and unblock the worker thread. - The query must be immediately removed from
QeProcessorImplandinformation_schema.active_queries. - Workload group slots must be promptly freed for queued queries.
How to Reproduce?
- Create a workload group with strict concurrency:
CREATE WORKLOAD GROUP wg_test PROPERTIES ('max_concurrency'='1', 'max_queue_size'='10', 'queue_timeout'='60000');
- Assign
wg_testto usertest. - Submit a query that takes several seconds (or simulate a lock wait / slow scan on BE).
- Abruptly kill the client process (send TCP FIN/RST) within 2 seconds before any result row is returned.
- Inspect
information_schema.active_queries:- The query remains
RUNNINGlong afterquery_timeout.
- The query remains
- Submit a second query from a new connection:
- The new query stays in
WAIT_IN_QUEUEand eventually times out.
- The new query stays in
Anything Else?
Observed in production where point-lookups (SELECT ... FROM tbl WHERE user_id = ...) encountering lock wait were abandoned by client microservices after 3s. The orphaned queries lived for >57 minutes in active_queries while subsequent queries piled up in WAIT_IN_QUEUE.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
Doris-Breakwater commentedon Sep 18, 2026
Breakwater-GitHub-Analysis-Slot: slot_e692bb511ebb
Initial assessment
This is a valid FE connection-teardown/cancellation gap, but the reported “indefinite query after
query_timeout” mechanism is only partially established. A disconnected MySQL session can leave its active query running until an execution-side timeout or another terminal event. That is enough to retain a workload-group slot unnecessarily and is worth fixing. However, static inspection does not support the stronger conclusion that removal fromconnectionMapby itself bypasses every query timeout.I checked the current heads named by the report:
masterat44e3ae2b,branch-4.1at8d2018bd, andbranch-3.0at4090f6c4. The relevant teardown structure is present in all three.Confirmed from code
unregisterConnection(context)(AcceptListener).ConnectPoolMgr.timeoutChecker()iterates onlyconnectionMap.values(), whileunregisterConnection()closes a transaction and removes the context without cancelling its active executor (timeout scan and unregister). Therefore, after the close callback wins the race, that context no longer receivesConnectContext.checkTimeout()calls. This part of the report is correct.Coordinator.cancel()cancels the queue token and invokes asynchronous BE-fragment cancellation (source, RPC path). When the worker unwinds,Coordinator.close()releases the queue token (source), and normal query finalization removes the query fromQeProcessorImpl(source). The missing link is invoking cancellation when the transport closes.coordBase.getNext()waits, so losing the client socket need not produce anEPIPEthat wakes the worker. The queue token remains held until the query exits and thefinallyblock closes the coordinator (source, cleanup).Important qualification
The ordinary BE-result path also enforces an absolute execution deadline independently of
TimeoutChecker:CoordinatorderivestimeoutDeadlinefromqueryOptions.execution_timeoutand gives it to eachResultReceiver(source);ResultReceiver.getNext()uses that deadline in its timed future wait and returns a timeout status (source). Waiting for a workload-group token is likewise bounded by the smaller applicable queue/execution timeout (source).Consequently, “the context was removed from
connectionMap, therefore thisgetNext()can run forever past its effective execution timeout” is not yet proven. The observed >3,400-second lifetime may reveal another blocked path or timeout discrepancy, but it needs runtime evidence. Also, although the production SQL is described as a point lookup, the short-circuitPointQueryExecutorbranch does not register a coordinator inQeProcessorImpl(source); that path cannot directly explain the reportedactive_queriesrow plus workload-group token. AnEXPLAIN/profile is needed to identify the actual execution path.Information needed to confirm the long-lived symptom
Please add:
active_queriessnapshots, plus the effective session/userquery_timeout, anySET_VARoverride, and the profile/query-options execution timeout for that same query.get result timeout,kill query timeout,Cancel execution, and deregistration messages.coordBase.getNext()alone does not distinguish the receiver, short-circuit, or another coordinator path.EXPLAINor the query profile. The current “slow query” step proves prompt disconnect cancellation only if the blocking condition is repeatable.Recommended next steps
CANCELLEDreason before removing/releasing session state. Keep this idempotent and race-safe. Avoid directly removing the query fromQeProcessorImplin the close callback; cancellation should wake the execution thread, whose existingfinallypath owns coordinator close, queue-token release, profile finalization, and query deregistration.ConnectContext.cancelQuery()reachesStmtExecutor.cancel(), which may wait for aCancelableCommand; the implementation should preserve prompt asynchronous transport teardown while still propagating coordinator/BE cancellation.ConnectPoolMgr.unregisterConnection(). Its current MySQL session-release hook is intentionally a no-op, so that refactor does not by itself close this gap.max_concurrency=1that closes the client during a deterministic BE wait and asserts: the first query is cancelled, the queue token is released, theactive_queriesrow disappears, and a second query starts without waiting for queue timeout. Cover both FIN and RST/dropped-client behavior and the race with normal query completion.Triage metadata: the issue currently has no labels and no assignee; the author is willing to submit a PR. Applying the repository’s FE/query-execution/workload-group bug ownership labels would help route it.
GJ100 commentedon Sep 18, 2026
Thanks for the thorough and helpful review!
Regarding the points raised:
Why it stayed alive for >3400s:
In our production environment, the effective
query_timeoutfor this database/user session was configured to 3600s (1 hour) (for occasional large ETL queries), while client microservices have a strict short network timeout of 3-5s.When a point-lookup or simple query encountered a lock wait / heavy compaction on BE, the client timed out at 3s and disconnected, but because of this gap, the orphaned query held the workload group slot for nearly an hour (approaching 3600s), starving all queued short queries in the same group!
Non-blocking cancellation in XNIO close listener:
Agreed! In our patch for
ConnectPoolMgr.unregisterConnection(), we will ensure cancellation is invoked withneedWaitCancelComplete = false(or asynchronously) so that the XNIO transport thread is never blocked.Coordination with PR [refactor](arrow-flight) One connection pool for MySQL connections and Arrow Flight SQL sessions #68101:
We will base our patch on current master and coordinate with PR [refactor](arrow-flight) One connection pool for MySQL connections and Arrow Flight SQL sessions #68101 to ensure the MySQL disconnect path properly triggers non-blocking cancellation before session teardown.
We will prepare the PR with the unit tests and non-blocking cancellation accordingly!