Skip to content

ARTEMIS-6355 perf client uses the wall clock of the other perf commands, and a negative latency no longer throws out of onMessage - #6873

Merged
clebertsuconic merged 1 commit into
apache:mainfrom
grepjava:ARTEMIS-6355
Nov 13, 2026
Merged

clebertsuconic merged 1 commit into
apache:mainfrom
grepjava:ARTEMIS-6355

Conversation

@grepjava

@grepjava grepjava commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

perf client stamps and reads message times with System.nanoTime(), while perf producer and perf consumer use the MicrosClock wall clock. When perf client consumes messages sent by perf producer, the latency comes out negative, the histogram throws out of onMessage, and the message is redelivered. The run stalls and never ends.

All three commands now take their clock from MicrosClock.timeProvider, including the fallback and its warning. It has to be the wall clock, as it's the only one that means anything across processes. perf client latencies are therefore measured on the wall clock now, like those of the other two commands.

Wall clocks can still differ between hosts or step back, so a negative latency is now recorded as 0 instead of throwing, for the receive latency and for the send latency.

MessageListenerBenchmarkTest checks a message stamped ahead of the receiver's clock and one stamped behind it. The first fails without the fix, with Histogram recorded value cannot be negative.

Tested against a broker built from main, changing only the CLI jar, with 100000 messages sent by perf producer --persistent --message-count 100000, then perf client --persistent --duration 20 --warmup 5 on the same queue:

  • Without the fix: still running at a 60 second timeout, with 16567 of these exceptions.
  • With the fix: ended normally with no exceptions, 225478 sent and 325478 received, which is the 100000 plus those sent.

https://issues.apache.org/jira/browse/ARTEMIS-6355

…ds, and a negative latency no longer throws out of onMessage
@clebertsuconic

Copy link
Copy Markdown
Contributor

I understand the change.. but I believe perf client is not supposed to run in parallel with producer and consumer, right?

Also: the new storage I'm working on will highly benefit from parallel queues, parallel producers... we should find a way to measure that.

@grepjava

grepjava commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

Not in parallel, no. It was sequential: perf producer filled a backlog and finished, then perf client started on the same queue, for the paging tests on #6838. It reads the producer's messages, stamped by the other clock, and the run never ends. That's why I used perf producer and perf consumer side by side for those tests, as they share the clock.

If you don't want to support that mix, the negative latency handling still seems worth having: wall clocks can differ between hosts, so perf producer and perf consumer on two hosts can hit the same exception, and a negative latency shouldn't cause a redelivery.

For parallel queues and producers, the connection tests I posted on #6838 are one way to measure it: perf client --num-destinations 16 --producers 1 --consumers 1 --threads 4 at 6, 12 and 24 connections. More connections made no difference on my box, at the default flush period or at 0.1 ms, but that's 4 shared CPUs with PostgreSQL on the same disk, so it can't show what a real database server would do. With #6870 in, --tx also lets the consumers commit in batches, so I can add that to the runs if it's useful.

@clebertsuconic
clebertsuconic merged commit 8440b71 into apache:main Nov 13, 2026
6 checks passed
@gemmellr

Copy link
Copy Markdown
Member

I'm not sure about this change for a few reasons.

Switching perf-client to the wall clock time source (default millisecond without-flags) when it is always expected to be sending to itself, now essentially requires all users interested in latency to set the opens flag to get access to the microsecond clock source for more useful latency results unlike before, which would potentially still not be as good as what it was using by default prior to the change, and is subject to changing backwards and causing 'negative latency' unlike before this change. It seems more useful to users by default, the way it was.

Allowing mixing previously-sent messages from perf producer with subsequent consumption by perf client on the same queue, does not seem like it generally makes sense anyway, particularly for latency as the latency reported would all be entirely skewed by the execution gap. The idea of having the distinct client / consumer+producer commands is essentially to use one or the other.

Negative latency seems like an error or problem worth at least flagging rather than silently ignoring, to make clear the results are bogus. If just saying it is 0 instead is then actually used as an effective 0 result, that will mean the latency result is really just incorrect anyway.

@grepjava

Copy link
Copy Markdown
Contributor Author

Thanks Robbie. On the third point I agree: clamping to 0 hides a bad result. I'm happy to follow up so a negative latency isn't recorded but counted, with the summary warning that the latency results are invalid.

On the clock, two things. The artemis.profile and artemis-utility.profile an instance is created with already pass --add-opens java.base/jdk.internal.misc=ALL-UNNAMED. So <instance>/bin/artemis perf client gets the microsecond wall clock without extra flags. Only the home bin/artemis falls back to milliseconds, with the warning, as perf producer and perf consumer always have. And nanoTime only means something within one JVM. Any message perf client didn't send itself in that process, such as a leftover from an earlier or interrupted run on a durable queue, gives a meaningless latency. Before #6873, a negative one threw out of onMessage and the message redelivered, so the run never ended (16567 exceptions in 60 seconds in my test).

That said, I take the point that the wall clock can step back where nanoTime can't. If you'd rather perf client went back to nanoTime, the follow-up could do that too, since the count-and-warn handling would still stop it stalling on stray messages. Which would you prefer?

@gemmellr

Copy link
Copy Markdown
Member

Yep, I realise nanotime is only useful inside a single JVM. I dont think thats a problem since messages from perf client are themselves really only useful in a single execution, for much the same reason it typically wouldnt make sense to combine perf producer and perf client usage on the same messages between executions.

I hadnt realised that the utility profile set the opens flag, which I guess somewhat negates that usability issue, at least for 'broker instance' usage (at least until such time as we ever stop doing that or it isnt allowed in future) but not the 'non instance' case.

It does still seem worse that perf client is subject to wall clock change when it wasnt before, and that 'no instance' usage of perf client would start dropping to ms precision. I think I'd prefer to put it back to nanoTime as it was.

@grepjava

Copy link
Copy Markdown
Contributor Author

Fair enough, thanks. I've opened #6909, which puts perf client back on nanoTime and replaces the clamp to 0 with a count: a negative latency is no longer recorded, and the summary (and the JSON summary) says how many there were and that the latency results are not valid. perf producer and perf consumer keep the wall clock, as before.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants