Repository navigation
Conversation
A stateful firewall *CAN* silently kill a pooled HTTP2 connection at L3 (no RST, no FIN). When this happens with okhttp, the next frame write that fills the kernel TCP send buffer blocks indefinitely. pingInterval is documented to detect exactly this, but three issues prevented it from firing: * The ping watchdog ran on writerQueue, so it queued behind the wedged frame write and never ticked. Moved to its own pingQueue; the ping write itself is fire-and-forget onto writerQueue. * failConnection -> close -> shutdown -> writer.goAway() deadlocked on writer.lock held by the wedged write. failConnection now cancels the socket first to unblock the writer. * SSLSocket.close() can block for many seconds attempting to send close_notify on a half-open connection. ConnectPlan now closes the raw TCP socket before the SSL layer on cancel. This PR adds a container-tests repro using socat amd an iptables DROP to simulate such a firewall drop. It asserts the failure within 3x pingInterval.
3.19 is gone
… the connection.
Restores the contract asserted by HttpOverHttp2Test.missingPongsFailsConnection.
The earlier dead-socket fix made failConnection() cancel the socket before calling close(), to unblock a writer stuck on a half-open TCP send. That ordering also surfaced SocketException("Socket closed") to in-flight callers instead of the StreamResetException(PROTOCOL_ERROR) they previously saw, because the reader thread's socket read returns SocketException before close() -> stream.close() has a chance to set errorCode on Http2Stream.
So: mark active streams with closeLater(PROTOCOL_ERROR) up front, before cancelling the socket.
closeLater enqueues the RST_STREAM via writerQueue rather than writing it synchronously under writer.lock — that lock is held by the stuck frame write, so close()'s synchronous RST_STREAM path would itself block.
With errorCode set then Http2Stream.closeInternal returns early on subseqent calls and a racing failConnection from the reader thread can not overwrite the callers StreamResetException with the SocketException it caught.
|
Great analysis. A permanently wedged writer is likely gonna cause a bunch of liveness bugs. I'll try to address this report specifically and that issue generally. |
|
@swankjesse any chance we could get this reviewed please? lmk if you need me to action any further...! |
| if (streams.isNotEmpty()) streams.values.toTypedArray() else null | ||
| } | ||
| streamsToClose?.forEach { stream -> | ||
| stream.closeLater(ErrorCode.PROTOCOL_ERROR) |
There was a problem hiding this comment.
I think this clears the exception
/**
* Abnormally terminate this stream. This enqueues a `RST_STREAM` frame and returns immediately.
*/
fun closeLater(errorCode: ErrorCode) {
if (!closeInternal(errorCode, null)) {
return // Already closed.
}
connection.writeSynResetLater(id, errorCode)
}
/** Returns true if this stream was closed. */
private fun closeInternal(
errorCode: ErrorCode,
errorException: IOException?,
): Boolean {
...
this.errorException = errorException
There was a problem hiding this comment.
Thanks @yschimke: two regressions and a test gap plugged and pushed.
| */ | ||
| @Isolated | ||
| @Timeout(value = 30, unit = TimeUnit.SECONDS) | ||
| class DeadSocketTest { |
There was a problem hiding this comment.
These no longer run, they were moved out to keep CI focused.
| settingsListenerQueue.shutdown() | ||
| } | ||
|
|
||
| private fun failConnection(e: IOException?) { |
There was a problem hiding this comment.
Maybe not a blocker and maybe not a huge concern, but this async work does feel racy, so I wonder if certain classes of errors might surface in two different ways now.
A stateful firewall CAN silently kill a pooled HTTP2 connection at L3 (no RST, no FIN).
When this happens with okhttp, the next frame write that fills the kernel TCP send buffer blocks indefinitely.
pingInterval is documented to detect exactly this, but three issues prevented it from firing:
The ping watchdog ran on writerQueue, so it queued behind the wedged frame write and never ticked. Moved to its own pingQueue; the ping write itself is fire-and-forget onto writerQueue.
failConnection -> close -> shutdown -> writer.goAway() deadlocked on writer.lock held by the wedged write. failConnection now cancels the socket first to unblock the writer.
SSLSocket.close() can block for many seconds attempting to send close_notify on a half-open connection. ConnectPlan now closes the raw TCP socket before the SSL layer on cancel.
This PR adds a container-tests repro using socat amd an iptables DROP to simulate such a firewall drop. It asserts the failure within 3x pingInterval.
(Apologies for additional pushes, was unable to run entire suite locally).