Repository navigation
ARTEMIS-6370 AMQP large message delivery loop on short body - #6916
agagliardi wants to merge 1 commit into
Conversation
An AMQP large message whose stored body is smaller than the body size recorded for it put the broker into an unrecoverable retry loop. The streaming loop in AMQPLargeMessageWriter passed the result of LargeBodyReader.readInto straight into ByteBuf.writerIndex, so the -1 that a sequential file read returns at end of file raised an IndexOutOfBoundsException naming a writerIndex of -1. That reached ProtonServerSenderContext.reportDeliveryError, which removed the consumer and handed the reference back to the queue, so the next consumer to attach failed the same way and the queue logged AMQ222151 indefinitely while delivering nothing. A zero length read had the same cause and spun the loop instead of raising anything. The writer now checks the read before it is used and raises a dedicated ActiveMQAMQPLargeMessageBodyTooSmallException, and the sender sends that one condition to the dead letter address rather than retrying a delivery that can never complete. It goes through ServerConsumer reject so the reference leaves the consumer's delivering list before the link is torn down, because the teardown would otherwise return the message to the queue and the loop would survive. The link is still closed, as a partial transfer may already be on the wire and proton-j cannot abort one. AMQPSessionCallback.disconnect also dereferenced a protocol context that the preceding consumer close clears on the queue executor, so the teardown raised a NullPointerException of its own. It now checks for null the way hasCredits in the same class already does. Assisted-by: Claude Code (Opus 5) on behalf of Antonio Gagliardi
d6bbd0a to
8c8db7a
Compare
|
This seems like it also would affect the AMQPTunneledCoreLargeMessageWriter as well which is not addressed here and it opens the question of what the Core message handling does in this case, does it also spin or is there a difference that allows it to reject and move the message to the dead letter address? |
|
@agagliardi i had fixed situations that would lead to an empty message before in previous versions. what version of the broker did you see that issue happening? |
Good catch! |
Do you remember which JIRAs those earlier fixes were? |
|
@agagliardi involving divert / mirroring: |
|
@agagliardi this commit: |
|
This fix targets a message that is sitting on disk and gets corrupted via truncation, the code currently will go ahead and try to deliver it without checking that the disk backed size matches what was recorded. It looks at though Core is also affected here although it seems like it might just send the truncation as some of its checks are only validated via assertions. I've worked up a test for Core that does something similar to the AMQP test that seems to show the same issues: |
Fixes an unrecoverable AMQP large message delivery loop. Details in
ARTEMIS-6370; the short version:
AMQPLargeMessageWriterfedreadInto's end-of-file-1intoByteBuf.writerIndex, raisingIndexOutOfBoundsException: readerIndex: 0, writerIndex: -1; a0length read spun the loop instead.attached afterwards failed the same way — the queue stopped delivering until an operator failed the
broker over.
ActiveMQAMQPLargeMessageBodyExceptionand the sender dead-letters the message rather than retrying,mirroring the
ConversionExceptionbranch already inreportDeliveryError. It usesServerConsumer.rejectso the reference leaves the delivering list before the link teardown, whichwould otherwise put the message back on the queue.
SenderImpl.abort()is ano-op in proton-j 0.35.0.
AMQPSessionCallback.disconnect, which NPE'd on a protocol context that the precedingconsumer close clears asynchronously — the same race as ARTEMIS-5598, at a call site it missed.
New codes:
AMQ119301(the exception) andAMQ111016(WARN when the message is dead-lettered).Tests
Three unit tests in
AMQPLargeMessageWriterTestand a new end-to-endAmqpTruncatedLargeMessageBodyTest; all of them fail without the production changes. With the writerfix reverted, two of the unit tests fail with the exact exception from the field report and the third
hangs to its timeout, demonstrating the zero-read spin.
Claude Code on behalf of Antonio Gagliardi
🤖 Generated with Claude Code