[improve][broker] Add counter to topic stats for delayed messages which exceed TTL at publish time

[zjxxzjwang]

Motivation

Currently, Pulsar Broker lacks a metric to track instances where delayed messages exceed their TTL. This results in messages set with delayed delivery times exceeding the TTL expiring before being consumed by users, with no mechanism to detect this occurrence. Consequently, there is a significant risk of message loss.

Modifications

1. Add a new member variable Rate exceedTTLDelayMessage = new Rate() to PersistentTopic to track the number of delayed messages exceeding the TTL.
2. Within the message sending logic of handleSend, add a check specifically for delayed messages whose delay time exceeds the TTL. If exceeded, call exceedTTLDelayMessage.recordEvent().
3. In TopicStatsImpl, add the exceedTTLDelayMessages variable to retrieve and display the count of delayed messages exceeding the TTL.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[Denovo1998]

Left some comments.

[zjxxzjwang]

@Denovo1998 Hello, could you please help me review again

[zjxxzjwang]

@Denovo1998 Hello, could you please help me review again

[lhotari]

Please check the review comments.
In addition, please revisit the PR title. It's currently [improve][broker]Increase the number of delayed messages sent whose delay time exceeds the TTL statistical metric which is not very clear. Something like [improve][broker] Add counter to topic stats for delayed messages which exceed TTL at publish time would be more accurate.

[zjxxzjwang]

@lhotari Hello, I have made the modifications. Can you help me review them

[lhotari]

One side-effect of this change is that the message metadata would get parsed when TTL is set. It has a small performance overhead compared to the previous behavior. Therefore, it would be useful to be able to disable this feature completely.

[lhotari]

Currently, Pulsar Broker lacks a metric to track instances where delayed messages exceed their TTL. This results in messages set with delayed delivery times exceeding the TTL expiring before being consumed by users, with no mechanism to detect this occurrence. Consequently, there is a significant risk of message loss.

The implementation doesn't seem to match the goal of the change ("track instances where delayed messages exceed their TTL").
It currently target the case where the message has already been expired when it gets published. How common is that case? What TTL value are you using?