[feat][broker] PIP-469: Legacy-aware topic policies backend routing and metadata-store topic policies

[BewareMyPower]

pip: #25547

This PR reuses TopicPoliciesTest to test most functionality of the new metadata store based implementation, so it also introduces improvements on TopicPoliciesTest, which requires too much time to run before (100+ tests, where each test calls internalSetup and internalCleanup).

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Implements PIP-469 topic-policies backend routing to preserve legacy __change_events behavior while enabling a new metadata-store-backed topic policies implementation, and refactors existing topic policies tests to run significantly faster.

Changes:

• Add LegacyAwareTopicPoliciesService to route per-namespace operations to either legacy system-topic or configured backend.
• Introduce MetadataStoreTopicPoliciesService implementation backed by metadata stores, with listener support.
• Refactor TopicPoliciesTest lifecycle to reduce repeated broker setup/teardown and add a derived test suite for the metadata-store backend.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/TopicPolicyTestUtils.java	Extends bypass-cache helper to work with legacy-aware routing and the metadata-store backend.
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/LegacyAwareTopicPoliciesServiceTest.java	Adds upgrade/downgrade and listener behavior coverage for legacy-aware routing and metadata-store storage.
pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/TopicPoliciesTest.java	Refactors test lifecycle to reduce runtime and adapts tests to multiple policies backends.
pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/MetadataStoreTopicPoliciesTest.java	Reuses TopicPoliciesTest against the metadata-store backend, disabling inapplicable system-topic tests.
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java	Loosens visibility on bundle-ownership cleanup to enable reuse/integration.
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/MetadataStoreTopicPoliciesService.java	Adds new metadata-store-backed implementation for topic policies with store notifications and listeners.
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/LegacyAwareTopicPoliciesService.java	Adds per-namespace routing layer between legacy system-topic backend and configured backend.
pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java	Wraps configured topic policies service with legacy-aware routing when system topics are enabled.
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java	Updates configuration documentation to describe built-in topic policies service implementations and legacy behavior.

[BewareMyPower]

We might need to consider whether to change the registerListener API because we cannot register listener for both LegacyAwareTopicPoliciesService and SystemTopicBasedTopicPoliciesService.