PIP-476: Extensible Key_Shared Dispatcher Mechanism

pip/pip-476.md

@@ -0,0 +1,108 @@
+# PIP-476: Extensible Key_Shared Dispatcher Mechanism
+
+# Background Knowledge
+
+In Pulsar Broker, message dispatching for Key_Shared subscriptions is implemented by `PersistentStickyKeyDispatcherMultipleConsumers` (which extends `PersistentDispatcherMultipleConsumers`).
+
+Since Pulsar 4.0, there is already a toggle between "Classic vs. new" implementations (`subscriptionKeySharedUseClassicPersistentImplementation` boolean config), but the creation logic is hardcoded in `PersistentSubscription.reuseOrCreateDispatcher()` and does not support loading third-party implementations.
+
+Furthermore, the key extension point methods in `PersistentStickyKeyDispatcherMultipleConsumers` (`filterAndGroupEntriesForDispatching`, `canDispatchEntry`) are `private`, and the constructor is package-private, making it impossible for subclasses to extend from outside the package.
+
+# Motivation
+
+The community consensus reached during the PIP-474 discussion is that new dispatching strategies (such as Hot Key Overflow) should be developed and validated as **standalone Dispatcher implementations**, rather than being embedded into the core class as conditional branches. This requires addressing three prerequisites:
+
+1. **Method visibility**: Core methods are `private`, preventing subclass overrides
+2. **Instantiation entry point**: `PersistentSubscription` has hardcoded creation logic, making it impossible to load third-party implementations
+3. **Lifecycle awareness**: Custom Dispatchers may manage additional resources (e.g., auxiliary ManagedLedgers) that need cleanup upon subscription deletion or topic unloading
+
+# Goals
+
+## In Scope
+
+- Change key methods and constructor of `PersistentStickyKeyDispatcherMultipleConsumers` to `protected`, allowing subclass extension
+- Add a broker configuration option to load custom Key_Shared Dispatcher implementations via reflection
+- Add lifecycle callbacks (default methods) to the `Dispatcher` interface, enabling custom implementations to react to subscription deletion and topic unloading events
+- Keep default behavior completely unchanged
+
+## Out of Scope
+
+- Changing the internal dispatching logic of existing Dispatchers
+- Providing stable internal API compatibility guarantees for third-party Dispatchers (marked as experimental in the initial phase)
+- Dispatcher extensibility for Shared / Exclusive / Failover subscription types (can be addressed as follow-up work)
+
+# High Level Design
+
+This PIP consists of three minimal changes:
+
+## 1. Method Visibility Adjustment
+
+Change the following members in `PersistentStickyKeyDispatcherMultipleConsumers` from `private` / package-private to `protected`:
+
+- **Constructor** — allows subclasses to extend from outside the package
+- **`filterAndGroupEntriesForDispatching()`** — core logic for message filtering and grouping; subclasses can override to implement custom dispatching strategies
+- **`canDispatchEntry()`** — per-entry dispatch decision; subclasses can override to add custom interception logic
+
+The following methods are already `protected` and together form the overridable extension points for subclasses: `trySendMessagesToConsumers()`, `isNormalReadAllowed()`, `getMaxEntriesReadLimit()`, `handleNormalReadNotAllowed()`.
+
+## 2. Custom Dispatcher Configuration
+
+Add a new configuration option `subscriptionKeySharedDispatcherClassName` (String, default empty). When non-empty, `PersistentSubscription.reuseOrCreateDispatcher()` loads the specified class via reflection, taking priority over the existing boolean toggle. The custom class must extend `PersistentStickyKeyDispatcherMultipleConsumers` and provide a constructor with the same signature as the base class.
+
+## 3. Lifecycle Callbacks
+
+Add two default methods to the `Dispatcher` interface:
+
+- **`onSubscriptionDeleted()`** — called when a subscription is deleted, for cleaning up persistent resources (e.g., auxiliary ManagedLedgers)
+- **`onTopicUnload()`** — called when a topic is unloaded from a broker, for releasing local resources (should NOT delete persistent state, as the topic may be reloaded on another broker)
+
+The default implementations are no-ops, with no impact on existing Dispatchers.
+
+## Public-facing Changes
+
+### Configuration
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `subscriptionKeySharedDispatcherClassName` | String | `""` (empty, uses built-in implementation) | Fully qualified class name of a custom Key_Shared Dispatcher implementation. Experimental feature. |
+
+`dynamic = true`, can be adjusted at runtime (only affects newly created subscriptions).
+
+### Public API / Binary Protocol / CLI / Metrics
+
+No additions.
+
+# Security Considerations
+
+The security model is consistent with the existing `topicFactoryClassName`: only broker administrators can modify the configuration, and the class must be on the broker classpath.
+
+# Backward & Forward Compatibility
+
+## Upgrade
+
+- Default configuration is empty, behavior is fully consistent with the current version
+- `private` → `protected` is a binary-compatible change
+- Lifecycle callbacks are `default` methods, no breakage to existing implementations
+
+## Downgrade / Rollback
+
+- After downgrade, the configuration is ignored and the built-in implementation is automatically used
+- Persistent resources created by custom Dispatchers may become orphaned and require manual cleanup
+
+## Geo-Replication
+
+Dispatchers are a broker-local mechanism and do not participate in Geo-Replication. Each region can be configured independently.
+
+# Alternatives
+
+# General Notes
+
+- This PIP is marked as **experimental** and may have incompatible changes in the future
+- Total change footprint is approximately 50 lines, with minimal rollback risk
+- This PIP is a prerequisite for PIP-474 (Key_Shared Hot Key Overflow), enabling PIP-474 to be developed and production-validated as a standalone Dispatcher implementation
+
+# Links
+
+* Mailing List discussion thread:
+* Mailing List voting thread:
+* Related: [PIP-474 PR #25706 discussion](https://github.com/apache/pulsar/pull/25706)