[26.1] Ordered rack/region preference for leader pinning#1598
[26.1] Ordered rack/region preference for leader pinning#1598kbatuigas wants to merge 9 commits intov-WIP/26.1from
Conversation
✅ Deploy Preview for redpanda-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
|
||
| * Enable config_ref:enable_rack_awareness,true,properties/cluster-properties[`enable_rack_awareness`] to distribute replicas across racks automatically. | ||
| * Set config_ref:default_leaders_preference,true,properties/cluster-properties[`default_leaders_preference`] before creating topics, so that rack-aware replica placement aligns with the preferred racks. | ||
| * Use xref:manage:cluster-maintenance/cluster-balancing.adoc[partition balancing] to move existing replicas to preferred racks. |
There was a problem hiding this comment.
Is this sufficient or should we link/mention a specific kind of balancing listed on this page? https://deploy-preview-1598--redpanda-docs-preview.netlify.app/current/manage/cluster-maintenance/cluster-balancing/
| = Leader Pinning | ||
| :description: Learn about leader pinning and how to configure a preferred partition leader location based on cloud availability zones or regions. | ||
| :description: Learn about Leader Pinning and how to configure a preferred partition leader location based on cloud availability zones or regions. | ||
| :page-topic-type: how-to |
There was a problem hiding this comment.
Note that we'll also need to add these attributes to https://github.com/redpanda-data/cloud-docs/blob/main/modules/develop/pages/produce-data/leader-pinning.adoc cc @micheleRP
| To prevent this scenario: | ||
|
|
||
| * Enable config_ref:enable_rack_awareness,true,properties/cluster-properties[`enable_rack_awareness`] to distribute replicas across racks automatically. | ||
| * Set config_ref:default_leaders_preference,true,properties/cluster-properties[`default_leaders_preference`] before creating topics, so that rack-aware replica placement aligns with the preferred racks. |
There was a problem hiding this comment.
This is not correct. The default leader preference will not influence the replica placement. As of right now there is no automatic solution for this scenario.
It is possible to place the replicas manually, though there is no guarantee that a balancer action will not move it later.
There was a problem hiding this comment.
@joe-redpanda Updated - is the manual reassignment worth mentioning, like so? https://github.com/redpanda-data/docs/pull/1598/changes#diff-09b2e0ca15e9a9e4ce3b4f194471533ebd3fcb2069c7393dac28d6e138cc5a2fR121-R122 And is it done using rpk cluster partitions move?
| // tag::single-source[] | ||
|
|
||
| Produce requests that write data to Redpanda topics go through the topic partition leader, which syncs messages across its follower replicas. For a Redpanda cluster deployed across multiple availability zones (AZs), leader pinning ensures that a topic's partition leaders are geographically closer to clients, which helps decrease networking costs and guarantees lower latency. | ||
| Produce requests that write data to Redpanda topics route through the topic partition leader, which syncs messages across its follower replicas. For a Redpanda cluster deployed across multiple availability zones (AZs), Leader Pinning ensures that a topic's partition leaders are geographically closer to clients, which helps decrease networking costs and guarantees lower latency. |
There was a problem hiding this comment.
| Produce requests that write data to Redpanda topics route through the topic partition leader, which syncs messages across its follower replicas. For a Redpanda cluster deployed across multiple availability zones (AZs), Leader Pinning ensures that a topic's partition leaders are geographically closer to clients, which helps decrease networking costs and guarantees lower latency. | |
| Produce requests that write data to Redpanda topics are routed through the topic partition leader, which syncs messages across its follower replicas. For a Redpanda cluster deployed across multiple availability zones (AZs), Leader Pinning ensures that a topic's partition leaders are geographically closer to clients, which helps decrease networking costs and guarantees lower latency. |
There was a problem hiding this comment.
Also, @kbatuigas can you add a glossterm for Leader Pinning and link to it from here? thx
| Produce requests that write data to Redpanda topics route through the topic partition leader, which syncs messages across its follower replicas. For a Redpanda cluster deployed across multiple availability zones (AZs), Leader Pinning ensures that a topic's partition leaders are geographically closer to clients, which helps decrease networking costs and guarantees lower latency. | ||
|
|
||
| If consumers are located in the same preferred region or AZ for leader pinning, and you have not set up xref:develop:consume-data/follower-fetching.adoc[follower fetching], leader pinning can also help reduce networking costs on consume requests. | ||
| If consumers are located in the same preferred region or AZ for Leader Pinning, and you have not set up xref:develop:consume-data/follower-fetching.adoc[follower fetching], Leader Pinning can also help reduce networking costs on consume requests. |
There was a problem hiding this comment.
Thinking maybe you should also mention Cloud Topics, as this is another way to reduce networking costs (at a slight expense to latency, but great for logs, data not reliant on speed).
| == Set leader rack preferences | ||
|
|
||
| You can use both a topic configuration property and a cluster configuration property to configure leader pinning. | ||
| You can use both a topic configuration property and a cluster configuration property to configure Leader Pinning. |
There was a problem hiding this comment.
Is this an either/or? You can configure Leader Pinning using either a topic configuration property or cluster configuration property? Or, do you have to configure both the topic and cluster configs?
There was a problem hiding this comment.
Why is the explanation for this sentence in a separate paragraph? Maybe keep together so people won't ask the same questions I asked above?
| This configuration is based on the following scenario: you have Redpanda deployed in a multi-AZ or multi-region cluster, and you have configured each broker so that the config_ref:rack,true,properties/broker-properties[] configuration property contains racks corresponding to the AZs: | ||
|
|
||
| * Set the topic configuration property xref:reference:properties/topic-properties.adoc#redpandaleaderspreference[`redpanda.leaders.preference`]. The property accepts the following string values: | ||
| * Set the topic configuration property xref:reference:properties/topic-properties.adoc#redpanda-leaders-preference[`redpanda.leaders.preference`]. The property accepts the following string values: |
There was a problem hiding this comment.
| * Set the topic configuration property xref:reference:properties/topic-properties.adoc#redpanda-leaders-preference[`redpanda.leaders.preference`]. The property accepts the following string values: | |
| * Set the topic configuration property xref:reference:properties/topic-properties.adoc#redpanda-leaders-preference[`redpanda.leaders.preference`]. This property accepts the following string values: |
| This property inherits the default value from the cluster property `default_leaders_preference`. | ||
| + | ||
| To find the rack identifiers of all brokers, run: | ||
| + | ||
| [,bash] | ||
| ---- | ||
| rpk cluster info | ||
| ---- | ||
| + | ||
| To set the topic property: | ||
| + | ||
| [,bash] | ||
| ---- | ||
| rpk topic alter-config <topic-name> --set redpanda.leaders.preference=ordered_racks:<rack1>,<rack2> | ||
| ---- |
There was a problem hiding this comment.
Since this is part of the topic config, shouldn't it all be indented under that bullet?
There was a problem hiding this comment.
Here's appears it looks in Cloud, which looks correct:
| + | ||
| [,bash] | ||
| ---- | ||
| rpk cluster config set default_leaders_preference ordered_racks:<rack1>,<rack2> |
There was a problem hiding this comment.
hmmm, why is this set/formatted differently from
rpk topic alter-config <topic-name> --set redpanda.leaders.preference=ordered_racks:<rack1>,<rack2>?
rpk cluster config set default_leaders_preference ordered_racks:<rack1>,<rack2>
|
|
||
| ifdef::env-cloud[] | ||
| == Configure leader pinning | ||
| == Set leader rack preferences |
There was a problem hiding this comment.
FYI--the learning objectives are not rendering in the Cloud output:
| * A topic with `A` as the preferred leader AZ will have its partition leaders uniformly distributed across B and C. | ||
| * A topic with `A,B` as the preferred leader AZs will have its partition leaders in B. | ||
| * A topic with `B` as the preferred leader AZ will have its partition leaders in B as well. |
There was a problem hiding this comment.
| * A topic with `A` as the preferred leader AZ will have its partition leaders uniformly distributed across B and C. | |
| * A topic with `A,B` as the preferred leader AZs will have its partition leaders in B. | |
| * A topic with `B` as the preferred leader AZ will have its partition leaders in B as well. | |
| * The topic with `A` as the preferred leader AZ will have its partition leaders uniformly distributed across B and C. | |
| * The topic with `A,B` as the preferred leader AZs will have its partition leaders in B. | |
| * The topic with `B` as the preferred leader AZ will have its partition leaders in B as well. |
There was a problem hiding this comment.
At first it was confusing to see the init "A"...especially since it is also the name of one of the AZs.
| For a topic configured with `ordered_racks:A,B,C`: | ||
|
|
||
| If there are three AZs: A, B, and C, and A becomes unavailable, the failover behavior is as follows: | ||
| * A topic with `A` as the first-priority rack will have its partition leaders in A. |
There was a problem hiding this comment.
| * A topic with `A` as the first-priority rack will have its partition leaders in A. | |
| * The topic with `A` as the first-priority rack will have its partition leaders in A. |
| * A topic with “B” as the preferred leader AZ will have its partition leaders in B as well. | ||
| If a higher-priority rack recovers and the topic's replication factor ensures that rack receives a replica, Redpanda automatically moves leaders back to the highest available preferred rack. | ||
|
|
||
| == Suggested reading |
| | Corresponding cluster property | ||
| | xref:reference:cluster-properties.adoc#default_leaders_preference[default_leaders_preference] |
There was a problem hiding this comment.
This does not render correctly--should have xref:reference:cluster-properties.adoc#default_leaders_preference[default_leaders_preference], no?
4 participants
Description
This pull request updates the documentation for Leader Pinning in Redpanda, introducing support for the new
ordered_racksconfiguration value and clarifying how Leader Pinning works, its limitations, and failover behavior. The changes enhance the explanation of how to configure Leader Pinning, especially for multi-region or multi-AZ deployments, and add guidance to avoid common pitfalls.Key changes include:
Leader Pinning feature enhancements:
ordered_racksconfiguration, which allows specifying preferred racks in priority order for partition leaders. This enables explicit failover sequencing for disaster recovery scenarios and is supported in Redpanda v26.1 and later. [1] [2]racks:(uniform distribution) andordered_racks:(priority-based failover) in both the main Leader Pinning docs and the release notes. [1] [2]Configuration and usage guidance:
Limitations and failover behavior:
racks:andordered_racks:under AZ/rack failures.These updates provide clearer, more actionable documentation for users configuring advanced leader placement strategies in Redpanda.
Resolves https://redpandadata.atlassian.net/browse/
Review deadline: 11 Mar
Page previews
Self-Managed - Leader Pinning
Topic properties >
redpanda.leaders.preferenceWhat's New
Cloud - Leader Pinning (need learning objectives added)
Checks