ARTEMIS-5573 and ARTEMIS-5975 Improve AMQP Size estimation by clebertsuconic · Pull Request #6323 · apache/artemis

clebertsuconic · 2026-03-27T01:46:15Z

No description provided.

tabish121

Overall looks good, nice simplification of some of the message handling

clebertsuconic · 2026-04-08T01:01:28Z

The reason I had to add a V4 is because of Paging:

On this part here, I encode the number of queues and the queueIDs:

artemis/artemis-server/src/main/java/org/apache/activemq/artemis/core/paging/impl/PagedMessageImpl.java

Lines 227 to 234 in 5b12277

    
           int queueIDsSize = buffer.readInt(); 
        
           queueIDs = new long[queueIDsSize]; 
        
           for (int i = 0; i < queueIDsSize; i++) { 
        
              queueIDs[i] = buffer.readLong(); 
        
           }

At the point of the encoder I don't have any reference to the number of bytes used by its own decoder.

I'm adding one on V4 now, if in the future anything else is added, we will stop reading at that marker.

clebertsuconic · 2026-04-08T01:45:26Z

I intend to squash the second commit on the first as soon some review is done on the new persister.

clebertsuconic · 2026-04-08T14:58:00Z

I'm setting it as ready to review. but this commit should be squashed before merged.

clebertsuconic · 2026-04-08T16:34:53Z

I needed to add versioning to AMQPLargeMessagePersister as well.

I could choose to skip memory calculations on large message... however I will prefer to keep it the same way as StandardMessage.

this is also better for future additions

clebertsuconic · 2026-04-14T17:10:17Z

I added a commit addressing the comments, let me know when I can squash it please

clebertsuconic · 2026-04-15T21:38:04Z

I added another commit to be squashed after review

gemmellr · 2026-04-16T16:27:26Z

+      // this is to allow us to determine the boundary of this persister, for future use.
+      buffer.writeInt(PERSISTER_SIZE);


I can see is that this would in future allows older brokers unaware of future changes to this new persister, to skip over anything added that they dont understand...but, do we actually send these 'persister encodings' over the wire at all? Wouldnt that make these changes already incompatible with existing versions if so? I kinda wonder if thats worth it if so, in order to write information thats already available via the message.

Either way, if we ever changed this v4 persister (or the v2 large) in future to do something else, and start writing a bigger value here so that old brokers can skip it, then surely we end up having to have the new broker inspect the size during decode and guess from any possible 'past variants of v4' what specific things were added 'without changing he persister [version]' are actually present or not in any given v4 (/v2 large) message encountered, purely from the size it finds?

That seems like it could be a bit of a PITA. Would it not be simpler long term to just have a basic list/map/array style structure so that we can always easily determine/look-up the values the given version understands and not need to be concerned with manually determining anything when additions are made? If we did that at the start then perhaps it would still be the V1 persister in use, unless we made some sort of drastic removal?

we already have extraProperties... but I don't want to use it here.. I wanted to keep it lean and just encoded

gemmellr · 2026-04-16T16:29:45Z

      }

      combinations.add(new Object[]{ARTEMIS_2_44_0, SNAPSHOT});
-      combinations.add(new Object[]{SNAPSHOT, ARTEMIS_2_44_0});


Is it expected there are now no combinations with a new main broker and old backup? There were 3 before which seems a bit of a switch up. EDIT: ah, so is that about persister incompatibility?

version compatibility support is alwasys forward . the backward compatibility is best effort.. and this case the journal persistence is broken by V4. the older version won't understand V4.

clebertsuconic · 2026-04-21T14:23:07Z

when I worked on this I wasn't planning to rewrite persistence for AMQP messages.

I will work on something for those persisters.

clebertsuconic · 2026-05-07T21:45:43Z

I am now using a MacCodec instead of a streamed bytes. I am not using a Hashmap to store records in between, but there's a codec that will store it like it was a hashMap.

I also added a clustering compatibility tests since the persistence is used with clustering.

- making it immutable to avoid races after updates like we had in the past - avoiding scanning after reloading by persisting extra data on storage

gemmellr

Gave it a skim over time, havent closely looked at most of it, but still had various comments so here they are. Most important one being, I do not think this should be in 2.54.0 at this stage.

gemmellr · 2026-05-08T14:43:37Z

   // 2.37.0
   public static final int ARTEMIS_2_37_0_VERSION = 136;

+   public static final int ARTEMIS_2_54_0_VERSION = 137;


I do not think this complex disk/wire level change should be targeting 2.54.0 at this late stage (it was already meant to be out).

I will rename to whatever comes next.. I used 54 at it's the next one now...

read it as TBD for now

gemmellr · 2026-05-08T15:38:10Z

+   private static final ThreadLocal<AMQPMessageMapCodec> threadLocal = ThreadLocal.withInitial(AMQPMessageMapCodec::new);
+
+   public static AMQPMessageMapCodec getInstance() {
+      AMQPMessageMapCodec reader = threadLocal.get();
+      return reader;
+   }


Possibly over-thinking it, but the this got me wondering if rather than a subclass, this would work nicer as a method that took a processor / callback-handler style argument. The persister would then just need 1 of those while it worked during startup and then it would go away when done, unlike this. You could also then separately test the codec class (current parent) and the processor/callback fully that way rather than e.g creating a test subclass to test the parent bits, and not really directly testing this class at all [it seems so far].

The large and regular persister could still use the same processor and then just toggle a flag to indicate which behaviour to give.

@gemmellr I can do that. although I'm using a thread local now because of the state I needed to store before creating a message.

There's one changed I wanted to make on the constructors for AMQPStandardMessage and AMQPLargeMessage, that I could keep the implementation stateless, and just post data directly on the message. then I wouldn't need a ThreadLocal and I could just work with the message directly.

not sure about a callback. I tried that before I sent the PR and I didn't like it.

clebertsuconic · 2026-05-08T18:32:12Z

@gemmellr I am marking this as draft because of the versioning.. whenever we decide to merge it I will change it and make it ready to commit

clebertsuconic force-pushed the message-size-amqp branch 8 times, most recently from f70cda5 to be90ac4 Compare March 29, 2026 21:44

clebertsuconic changed the title ~~ARTEMIS-TBD Improving memory estimates on AMQP~~ ARTEMIS-5573 and ARTEMIS-5975 Improve AMQP Size estimation and make it static Mar 29, 2026

clebertsuconic force-pushed the message-size-amqp branch 9 times, most recently from 0611d5a to 03b7176 Compare March 30, 2026 22:51

clebertsuconic marked this pull request as ready for review March 30, 2026 22:51

clebertsuconic force-pushed the message-size-amqp branch 8 times, most recently from 94efe35 to 70beb3a Compare April 1, 2026 13:07

clebertsuconic changed the title ~~ARTEMIS-5573 and ARTEMIS-5975 Improve AMQP Size estimation and make it static~~ ARTEMIS-5573 and ARTEMIS-5975 Improve AMQP Size estimation Apr 1, 2026

tabish121 self-requested a review April 1, 2026 14:14

tabish121 reviewed Apr 1, 2026

View reviewed changes

Comment thread ...mqp-protocol/src/main/java/org/apache/activemq/artemis/protocol/amqp/broker/AMQPMessage.java Outdated

Comment thread ...mqp-protocol/src/main/java/org/apache/activemq/artemis/protocol/amqp/broker/AMQPMessage.java Outdated

clebertsuconic force-pushed the message-size-amqp branch 6 times, most recently from 09bb86b to 540ad20 Compare April 8, 2026 00:58

clebertsuconic force-pushed the message-size-amqp branch 4 times, most recently from 1cb4403 to fa1128e Compare April 8, 2026 01:26

clebertsuconic commented Apr 8, 2026

View reviewed changes

Comment thread ...mqp-protocol/src/main/java/org/apache/activemq/artemis/protocol/amqp/broker/AMQPMessage.java Outdated

clebertsuconic commented Apr 8, 2026

View reviewed changes

Comment thread ...l/src/main/java/org/apache/activemq/artemis/protocol/amqp/broker/AMQPMessagePersisterV4.java Outdated

tabish121 reviewed Apr 8, 2026

View reviewed changes

Comment thread ...l/src/main/java/org/apache/activemq/artemis/protocol/amqp/broker/AMQPMessagePersisterV4.java Outdated

Comment thread .../main/java/org/apache/activemq/artemis/protocol/amqp/broker/AMQPLargeMessagePersisterV2.java Outdated

tabish121 reviewed Apr 8, 2026

View reviewed changes

Comment thread ...mqp-protocol/src/main/java/org/apache/activemq/artemis/protocol/amqp/broker/AMQPMessage.java Outdated

tabish121 reviewed Apr 13, 2026

View reviewed changes

Comment thread ...ocol/src/main/java/org/apache/activemq/artemis/protocol/amqp/broker/AMQPStandardMessage.java Outdated

gemmellr reviewed Apr 14, 2026

View reviewed changes

tabish121 reviewed Apr 15, 2026

View reviewed changes

Comment thread ...ocol/src/main/java/org/apache/activemq/artemis/protocol/amqp/broker/AMQPStandardMessage.java Outdated

gemmellr reviewed Apr 16, 2026

View reviewed changes

ARTEMIS-5573 and ARTEMIS-5975 Improve AMQP Size estimation

eb4ae56

- making it immutable to avoid races after updates like we had in the past - avoiding scanning after reloading by persisting extra data on storage

gemmellr reviewed May 8, 2026

View reviewed changes

ARTEMIS-5573 and ARTEMIS-5975 Addressin comments from pull request

c141421

		// this is to allow us to determine the boundary of this persister, for future use.
		buffer.writeInt(PERSISTER_SIZE);

Conversation

clebertsuconic commented Mar 27, 2026

Uh oh!

tabish121 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

clebertsuconic commented Apr 8, 2026

Uh oh!

Uh oh!

clebertsuconic commented Apr 8, 2026

Uh oh!

clebertsuconic commented Apr 8, 2026

Uh oh!

clebertsuconic commented Apr 8, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

clebertsuconic commented Apr 14, 2026

Uh oh!

Uh oh!

clebertsuconic commented Apr 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clebertsuconic commented Apr 21, 2026

Uh oh!

clebertsuconic commented May 7, 2026

Uh oh!

gemmellr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clebertsuconic commented May 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants