Skip to content

ARTEMIS-6370 AMQP large message delivery loop on short body - #6916

Open
agagliardi wants to merge 1 commit into
apache:mainfrom
agagliardi:ARTEMIS-6370-amqp-large-message-short-body-loop
Open

agagliardi wants to merge 1 commit into
apache:mainfrom
agagliardi:ARTEMIS-6370-amqp-large-message-short-body-loop

Conversation

@agagliardi

Copy link
Copy Markdown

Fixes an unrecoverable AMQP large message delivery loop. Details in
ARTEMIS-6370; the short version:

  • AMQPLargeMessageWriter fed readInto's end-of-file -1 into ByteBuf.writerIndex, raising
    IndexOutOfBoundsException: readerIndex: 0, writerIndex: -1; a 0 length read spun the loop instead.
  • The resulting delivery error removed the consumer and requeued the message, so every consumer that
    attached afterwards failed the same way — the queue stopped delivering until an operator failed the
    broker over.
  • A short body can never satisfy the recorded size, so this one condition now raises a dedicated
    ActiveMQAMQPLargeMessageBodyException and the sender dead-letters the message rather than retrying,
    mirroring the ConversionException branch already in reportDeliveryError. It uses
    ServerConsumer.reject so the reference leaves the delivering list before the link teardown, which
    would otherwise put the message back on the queue.
  • The link is still closed: a partial transfer may already be on the wire and SenderImpl.abort() is a
    no-op in proton-j 0.35.0.
  • Also null-guards AMQPSessionCallback.disconnect, which NPE'd on a protocol context that the preceding
    consumer close clears asynchronously — the same race as ARTEMIS-5598, at a call site it missed.

New codes: AMQ119301 (the exception) and AMQ111016 (WARN when the message is dead-lettered).

Tests

Three unit tests in AMQPLargeMessageWriterTest and a new end-to-end
AmqpTruncatedLargeMessageBodyTest; all of them fail without the production changes. With the writer
fix reverted, two of the unit tests fail with the exact exception from the field report and the third
hangs to its timeout, demonstrating the zero-read spin.

Claude Code on behalf of Antonio Gagliardi

🤖 Generated with Claude Code

An AMQP large message whose stored body is smaller than the body size
recorded for it put the broker into an unrecoverable retry loop. The
streaming loop in AMQPLargeMessageWriter passed the result of
LargeBodyReader.readInto straight into ByteBuf.writerIndex, so the -1
that a sequential file read returns at end of file raised an
IndexOutOfBoundsException naming a writerIndex of -1.

That reached ProtonServerSenderContext.reportDeliveryError, which
removed the consumer and handed the reference back to the queue, so
the next consumer to attach failed the same way and the queue logged
AMQ222151 indefinitely while delivering nothing. A zero length read
had the same cause and spun the loop instead of raising anything.

The writer now checks the read before it is used and raises a
dedicated ActiveMQAMQPLargeMessageBodyTooSmallException, and the
sender sends that one condition to the dead letter address rather
than retrying a delivery that can never complete. It goes through
ServerConsumer reject so the reference leaves the consumer's
delivering list before the link is torn down, because the teardown
would otherwise return the message to the queue and the loop would
survive. The link is still closed, as a partial transfer may already
be on the wire and proton-j cannot abort one.

AMQPSessionCallback.disconnect also dereferenced a protocol context
that the preceding consumer close clears on the queue executor, so
the teardown raised a NullPointerException of its own. It now checks
for null the way hasCredits in the same class already does.

Assisted-by: Claude Code (Opus 5) on behalf of Antonio Gagliardi
@agagliardi
agagliardi force-pushed the ARTEMIS-6370-amqp-large-message-short-body-loop branch from d6bbd0a to 8c8db7a Compare November 18, 2026 13:58
@tabish121

Copy link
Copy Markdown
Contributor

This seems like it also would affect the AMQPTunneledCoreLargeMessageWriter as well which is not addressed here and it opens the question of what the Core message handling does in this case, does it also spin or is there a difference that allows it to reject and move the message to the dead letter address?

@clebertsuconic

Copy link
Copy Markdown
Contributor

@agagliardi i had fixed situations that would lead to an empty message before in previous versions.

what version of the broker did you see that issue happening?

@agagliardi

Copy link
Copy Markdown
Author

This seems like it also would affect the AMQPTunneledCoreLargeMessageWriter as well which is not addressed here and it opens the question of what the Core message handling does in this case, does it also spin or is there a difference that allows it to reject and move the message to the dead letter address?

Good catch!
AMQPTunneledCoreLargeMessageWriter.tryDeliveryMessageBody() had the same unguarded read. I'm testing an updated PR to match it too.

@agagliardi

agagliardi commented Nov 20, 2026 •

Copy link
Copy Markdown
Author

@agagliardi i had fixed situations that would lead to an empty message before in previous versions.

what version of the broker did you see that issue happening?

2.53.0.redhat-00003. Same on the 2.33.0.PATCH.3307.

Do you remember which JIRAs those earlier fixes were?

@clebertsuconic

Copy link
Copy Markdown
Contributor

@agagliardi involving divert / mirroring:

commit 86ed6b4aab3ab1c601ee3414d46e247d44c14641
Author: Clebert Suconic <clebertsuconic@apache.org>
Date:   Mon Jul 6 16:09:22 2026 -0400

   ARTEMIS-6146 Large Message inconsistencies through Divert and Mirror

@clebertsuconic

Copy link
Copy Markdown
Contributor

@agagliardi this commit:
86ed6b4

@tabish121

Copy link
Copy Markdown
Contributor

This fix targets a message that is sitting on disk and gets corrupted via truncation, the code currently will go ahead and try to deliver it without checking that the disk backed size matches what was recorded. It looks at though Core is also affected here although it seems like it might just send the truncation as some of its checks are only validated via assertions. I've worked up a test for Core that does something similar to the AMQP test that seems to show the same issues:

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.activemq.artemis.tests.integration.largemessage;

import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.junit.jupiter.api.Assertions.fail;

import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.RandomAccessFile;

import org.apache.activemq.artemis.api.core.ActiveMQException;
import org.apache.activemq.artemis.api.core.QueueConfiguration;
import org.apache.activemq.artemis.api.core.RoutingType;
import org.apache.activemq.artemis.api.core.SimpleString;
import org.apache.activemq.artemis.api.core.client.ClientConsumer;
import org.apache.activemq.artemis.api.core.client.ClientMessage;
import org.apache.activemq.artemis.api.core.client.ClientProducer;
import org.apache.activemq.artemis.api.core.client.ClientSession;
import org.apache.activemq.artemis.api.core.client.ClientSessionFactory;
import org.apache.activemq.artemis.api.core.client.ServerLocator;
import org.apache.activemq.artemis.core.server.ActiveMQServer;
import org.apache.activemq.artemis.core.server.Queue;
import org.apache.activemq.artemis.core.server.impl.AddressInfo;
import org.apache.activemq.artemis.tests.util.ActiveMQTestBase;
import org.apache.activemq.artemis.tests.util.Wait;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.Timeout;

/**
 * Tests how the broker handles a Core large message whose body is truncated on disk after being
 * stored and then a client attaches to try and consume that message.
 * <p>
 * Core dispatch (ServerConsumerImpl.CoreLargeMessageDeliverer) streams the stored body in chunks,
 * using the body size recorded on the message as the loop bound, so correctness hinges on checking
 * what each read actually returned against that recorded size. If that check is missing or ever
 * weakened, a body that ends early on disk goes unnoticed in one of two ways: the consumer is
 * handed a "complete" message whose tail is bytes the broker never stored, or the delivery stalls
 * mid-stream and the message is left in flight indefinitely, blocking the queue behind it. This
 * test guards both outcomes: a message whose stored body ends before its recorded size must fail
 * delivery cleanly instead of fabricating a body, and it must be routed to the Dead Letter Address
 * rather than retried or held forever.
 */
public class CoreTruncatedLargeMessageBodyTest extends ActiveMQTestBase {

   private static final SimpleString QUEUE_NAME = SimpleString.of("TruncatedBodyTest");
   private static final SimpleString DEAD_LETTER_ADDRESS = SimpleString.of("ActiveMQ.DLQ");

   private static final int MIN_LARGE_MESSAGE_SIZE = 10 * 1024;
   private static final int PAYLOAD_SIZE = 100 * 1024;
   private static final int TRUNCATED_SIZE = 1024;

   @Test
   @Timeout(60)
   public void testTruncatedBodyIsNotFabricatedNorStuckForever() throws Exception {
      final ActiveMQServer server = createServer(true);
      server.start();

      server.addAddressInfo(new AddressInfo(DEAD_LETTER_ADDRESS, RoutingType.ANYCAST));
      server.createQueue(QueueConfiguration.of(DEAD_LETTER_ADDRESS).setRoutingType(RoutingType.ANYCAST));

      final ServerLocator locator = createInVMNonHALocator();
      locator.setMinLargeMessageSize(MIN_LARGE_MESSAGE_SIZE);
      // bounds the client wait on a body that may never arrive (default call timeout is 30s)
      locator.setCallTimeout(5000);

      final ClientSessionFactory sessionFactory = createSessionFactory(locator);
      final ClientSession session = addClientSession(sessionFactory.createSession(false, true, true));
      session.createQueue(QueueConfiguration.of(QUEUE_NAME).setRoutingType(RoutingType.ANYCAST));

      // Send with no consumer attached so the whole body is spooled to disk before anything reads it
      final ClientProducer producer = session.createProducer(QUEUE_NAME);
      final ClientMessage message = session.createMessage(true);

      // a non zero pattern: fabricated padding is never mistaken for the stored body
      final byte[] payload = new byte[PAYLOAD_SIZE];
      for (int i = 0; i < payload.length; i++) {
         payload[i] = getSamplebyte(i);
      }
      message.getBodyBuffer().writeBytes(payload);

      producer.send(message);
      producer.close();

      final Queue queue = server.locateQueue(QUEUE_NAME);
      final Queue deadLetterQueue = server.locateQueue(DEAD_LETTER_ADDRESS);

      Wait.assertEquals(1L, queue::getMessageCount, 5000, 100);

      truncateStoredBodies();

      // Attaching a consumer is what triggers the read of the body that is no longer all there
      final ClientConsumer consumer = session.createConsumer(QUEUE_NAME);
      session.start();

      final ClientMessage received = consumer.receive(10_000);

      if (received != null) {
         // Whatever was streamed over the wire, the application must not be handed a completed body
         // made of bytes the broker never stored for the message
         final ByteArrayOutputStream sink = new ByteArrayOutputStream();
         try {
            received.saveToOutputStream(sink);
            fail("a large message whose stored body was truncated was completed at the client with size: " + sink.size());
         } catch (ActiveMQException expected) {
            // the body never completed; acceptable only if the broker then gives up on the message
         }
      }

      // The body can never satisfy the recorded size, so the message has to leave the queue and be routed
      // to the Dead Letter Address to avoid sending truncated payload messages. Leaving it in flight blocks
      // this consumer for every following message and the next attach fails on it in the same way.
      Wait.assertEquals(0L, queue::getMessageCount, 5_000, 100);
      Wait.assertEquals(1L, deadLetterQueue::getMessageCount, 5_000, 100);
   }

   private void truncateStoredBodies() throws Exception {
      final File largeMessagesDir = new File(getLargeMessagesDir());
      final File[] bodies = largeMessagesDir.listFiles((dir, name) -> name.endsWith(".msg"));

      assertNotNull(bodies, "no large messages directory was created for the broker");
      assertTrue(bodies.length > 0, "expected the large message body to have been stored");

      for (File body : bodies) {
         try (RandomAccessFile file = new RandomAccessFile(body, "rw")) {
            file.setLength(TRUNCATED_SIZE);
         }
      }
   }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants