From 056398bb20706cb92de987508570b9101a76be21 Mon Sep 17 00:00:00 2001 From: micheleRP Date: Mon, 2 Mar 2026 13:56:32 -0700 Subject: [PATCH 1/9] DOC-1600 Document feature Fix cross region RRRs on AWS --- .../manage/partials/remote-read-replicas.adoc | 50 +++++++++++++++++-- 1 file changed, 47 insertions(+), 3 deletions(-) diff --git a/modules/manage/partials/remote-read-replicas.adoc b/modules/manage/partials/remote-read-replicas.adoc index ab0ee72961..d442422e9b 100644 --- a/modules/manage/partials/remote-read-replicas.adoc +++ b/modules/manage/partials/remote-read-replicas.adoc @@ -36,9 +36,9 @@ You need the following: * An origin cluster with xref:{tiered-storage-link}#set-up-tiered-storage[Tiered Storage] set up. Multi-region buckets or containers are not supported. * A topic on the origin cluster, which you can use as a Remote Read Replica topic on the remote cluster. * A separate remote cluster. -** AWS: The remote cluster must be in the same region as the origin cluster's storage bucket/container. +** AWS: The remote cluster can be in the same or a different region as the origin cluster's S3 bucket. For cross-region RRR topics, see <>. ** GCP: The remote cluster can be in the same or a different region as the bucket/container. -** Azure: Remote read replicas are not supported. +** Azure: Remote read replicas are not supported. include::shared:partial$enterprise-license.adoc[] @@ -56,7 +56,7 @@ You must configure access to the same object storage as the origin cluster. ifndef::env-kubernetes[] To set up a Remote Read Replica topic on a separate remote cluster: -. Create a remote cluster for the Remote Read Replica topic. For AWS, the remote cluster must be in the same region as the origin cluster's storage bucket/container. For GCP, the remote cluster can be in the same or a different region as the bucket/container. +. Create a remote cluster for the Remote Read Replica topic. For GCP, the remote cluster can be in the same or a different region as the bucket/container. For AWS, the remote cluster can be in the same or a different region, but cross-region RRR topics require additional configuration. See <>. . Run `rpk cluster config edit`, and then specify properties specific to your object storage provider (your cluster will require a restart after any changes to these properties): + @@ -437,6 +437,50 @@ rpk topic create -c redpanda.remote.readreplica= * Do not use `redpanda.remote.read` or `redpanda.remote.write` with `redpanda.remote.readreplica`. Redpanda ignores the values for remote read and remote write properties on read replica topics. ==== +[[create-cross-region-rrr-topic]] +=== Create a cross-region Remote Read Replica topic on AWS + +Use this configuration only when the remote cluster is in a *different AWS region* than the origin cluster's S3 bucket. For same-region AWS or GCP deployments, use the standard <>. + +==== Prerequisites + +The xref:reference:properties/object-storage-properties.adoc[`cloud_storage_url_style`] cluster property must be set explicitly to `virtual_host` or `path` on the remote cluster. The default value does not support cross-region Remote Read Replicas. + +==== Create the topic + +To create a cross-region Remote Read Replica topic, append `region` and `endpoint` query-string parameters to the bucket name: + +[,bash] +---- +rpk topic create \ + -c redpanda.remote.readreplica=?region=&endpoint=s3..amazonaws.com +---- + +Replace the following placeholders: + +- ``: The name of the original topic on the origin cluster. +- ``: The S3 bucket configured on the origin cluster (`cloud_storage_bucket`). +- ``: The AWS region of the origin cluster's S3 bucket (not the remote cluster's region). For example, `us-east-1`. + +For example, if the origin cluster stores data in a bucket called `my-bucket` in `us-east-1`: + +[,bash] +---- +rpk topic create my-topic \ + -c redpanda.remote.readreplica=my-bucket?region=us-east-1&endpoint=s3.us-east-1.amazonaws.com +---- + +NOTE: The `endpoint` value must not include the bucket name. Redpanda automatically prepends the bucket name when using `virtual_host` URL style. + +==== Limits + +Each unique combination of region and endpoint creates a dynamic object storage upstream on the remote cluster. A cluster supports a maximum of 10 dynamic upstreams. + +How upstreams are counted depends on `cloud_storage_url_style`: + +- `virtual_host`: Each unique combination of bucket, region, and endpoint counts as one upstream. You can create up to 10 distinct cross-region RRR topics per cluster. +- `path`: Each unique combination of region and endpoint counts as one upstream (the bucket name is not part of the key). You can create cross-region RRR topics for an unlimited number of buckets, as long as those buckets are spread across no more than 10 distinct region/endpoint combinations. + == Reduce lag in data availability :config-ref: cloud_storage_segment_max_upload_interval_sec From 6bbce24ac6341beb850b9e7fab14741007fcbb3f Mon Sep 17 00:00:00 2001 From: micheleRP Date: Mon, 2 Mar 2026 15:33:33 -0700 Subject: [PATCH 2/9] minor edits --- modules/manage/partials/remote-read-replicas.adoc | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/modules/manage/partials/remote-read-replicas.adoc b/modules/manage/partials/remote-read-replicas.adoc index d442422e9b..7e10d30ff3 100644 --- a/modules/manage/partials/remote-read-replicas.adoc +++ b/modules/manage/partials/remote-read-replicas.adoc @@ -36,8 +36,8 @@ You need the following: * An origin cluster with xref:{tiered-storage-link}#set-up-tiered-storage[Tiered Storage] set up. Multi-region buckets or containers are not supported. * A topic on the origin cluster, which you can use as a Remote Read Replica topic on the remote cluster. * A separate remote cluster. -** AWS: The remote cluster can be in the same or a different region as the origin cluster's S3 bucket. For cross-region RRR topics, see <>. -** GCP: The remote cluster can be in the same or a different region as the bucket/container. +** AWS: The remote cluster can be in the same region or a different region as the origin cluster's S3 bucket. For cross-region Remote Read Replica topics, see <>. +** GCP: The remote cluster can be in the same region or a different region as the bucket/container. ** Azure: Remote read replicas are not supported. include::shared:partial$enterprise-license.adoc[] @@ -56,7 +56,7 @@ You must configure access to the same object storage as the origin cluster. ifndef::env-kubernetes[] To set up a Remote Read Replica topic on a separate remote cluster: -. Create a remote cluster for the Remote Read Replica topic. For GCP, the remote cluster can be in the same or a different region as the bucket/container. For AWS, the remote cluster can be in the same or a different region, but cross-region RRR topics require additional configuration. See <>. +. Create a remote cluster for the Remote Read Replica topic. For GCP, the remote cluster can be in the same or a different region as the bucket/container. For AWS, the remote cluster can be in the same or a different region, but cross-region Remote Read Replica topics require additional configuration. See <>. . Run `rpk cluster config edit`, and then specify properties specific to your object storage provider (your cluster will require a restart after any changes to these properties): + @@ -444,7 +444,7 @@ Use this configuration only when the remote cluster is in a *different AWS regio ==== Prerequisites -The xref:reference:properties/object-storage-properties.adoc[`cloud_storage_url_style`] cluster property must be set explicitly to `virtual_host` or `path` on the remote cluster. The default value does not support cross-region Remote Read Replicas. +The xref:reference:properties/object-storage-properties.adoc#cloud_storage_url_style[`cloud_storage_url_style`] cluster property must be set explicitly to `virtual_host` or `path` on the remote cluster. The default value does not support cross-region Remote Read Replicas. ==== Create the topic @@ -478,8 +478,8 @@ Each unique combination of region and endpoint creates a dynamic object storage How upstreams are counted depends on `cloud_storage_url_style`: -- `virtual_host`: Each unique combination of bucket, region, and endpoint counts as one upstream. You can create up to 10 distinct cross-region RRR topics per cluster. -- `path`: Each unique combination of region and endpoint counts as one upstream (the bucket name is not part of the key). You can create cross-region RRR topics for an unlimited number of buckets, as long as those buckets are spread across no more than 10 distinct region/endpoint combinations. +- `virtual_host`: Each unique combination of bucket, region, and endpoint counts as one upstream. You can create up to 10 distinct cross-region Remote Read Replica topics per cluster. +- `path`: Each unique combination of region and endpoint counts as one upstream (the bucket name is not part of the key). You can create cross-region Remote Read Replica topics for an unlimited number of buckets, as long as those buckets are spread across no more than 10 distinct region/endpoint combinations. == Reduce lag in data availability From 1a3f394a72874ae8c9045fa39541c76a405318ff Mon Sep 17 00:00:00 2001 From: micheleRP Date: Mon, 2 Mar 2026 15:44:40 -0700 Subject: [PATCH 3/9] coderabbit fix --- modules/manage/partials/remote-read-replicas.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/manage/partials/remote-read-replicas.adoc b/modules/manage/partials/remote-read-replicas.adoc index 7e10d30ff3..cd17e08a17 100644 --- a/modules/manage/partials/remote-read-replicas.adoc +++ b/modules/manage/partials/remote-read-replicas.adoc @@ -36,8 +36,8 @@ You need the following: * An origin cluster with xref:{tiered-storage-link}#set-up-tiered-storage[Tiered Storage] set up. Multi-region buckets or containers are not supported. * A topic on the origin cluster, which you can use as a Remote Read Replica topic on the remote cluster. * A separate remote cluster. -** AWS: The remote cluster can be in the same region or a different region as the origin cluster's S3 bucket. For cross-region Remote Read Replica topics, see <>. -** GCP: The remote cluster can be in the same region or a different region as the bucket/container. +** AWS: The remote cluster can be in the same or a different region as the origin cluster's S3 bucket. For cross-region Remote Read Replica topics, see <>. +** GCP: The remote cluster can be in the same or a different region as the bucket/container. ** Azure: Remote read replicas are not supported. include::shared:partial$enterprise-license.adoc[] From 72b6eaa2d39073d4d6a152c5a0b5fd29d309ab88 Mon Sep 17 00:00:00 2001 From: Michele Cyran Date: Tue, 3 Mar 2026 09:49:00 -0700 Subject: [PATCH 4/9] Update modules/manage/partials/remote-read-replicas.adoc Co-authored-by: Nicolae Vartolomei --- modules/manage/partials/remote-read-replicas.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/manage/partials/remote-read-replicas.adoc b/modules/manage/partials/remote-read-replicas.adoc index cd17e08a17..b5393c1727 100644 --- a/modules/manage/partials/remote-read-replicas.adoc +++ b/modules/manage/partials/remote-read-replicas.adoc @@ -474,7 +474,7 @@ NOTE: The `endpoint` value must not include the bucket name. Redpanda automatica ==== Limits -Each unique combination of region and endpoint creates a dynamic object storage upstream on the remote cluster. A cluster supports a maximum of 10 dynamic upstreams. +Each unique combination of region and endpoint creates a dynamic object storage upstream on the remote cluster. A cluster supports a maximum of 10 active dynamic upstreams. How upstreams are counted depends on `cloud_storage_url_style`: From 03c75f94075bd4d993fd8fc90c1121712c1514cd Mon Sep 17 00:00:00 2001 From: micheleRP Date: Tue, 3 Mar 2026 09:59:45 -0700 Subject: [PATCH 5/9] incorporate Nicolae's feedback --- modules/manage/partials/remote-read-replicas.adoc | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/modules/manage/partials/remote-read-replicas.adoc b/modules/manage/partials/remote-read-replicas.adoc index b5393c1727..ead585dda3 100644 --- a/modules/manage/partials/remote-read-replicas.adoc +++ b/modules/manage/partials/remote-read-replicas.adoc @@ -448,7 +448,13 @@ The xref:reference:properties/object-storage-properties.adoc#cloud_storage_url_s ==== Create the topic -To create a cross-region Remote Read Replica topic, append `region` and `endpoint` query-string parameters to the bucket name: +To create a cross-region Remote Read Replica topic, append `region` and `endpoint` query-string parameters to the bucket name. + +In the following example, replace the placeholders: + +- ``: The name of the original topic on the origin cluster. +- ``: The S3 bucket configured on the origin cluster (`cloud_storage_bucket`). +- ``: The AWS region of the origin cluster's S3 bucket (not the remote cluster's region). [,bash] ---- @@ -456,12 +462,6 @@ rpk topic create \ -c redpanda.remote.readreplica=?region=&endpoint=s3..amazonaws.com ---- -Replace the following placeholders: - -- ``: The name of the original topic on the origin cluster. -- ``: The S3 bucket configured on the origin cluster (`cloud_storage_bucket`). -- ``: The AWS region of the origin cluster's S3 bucket (not the remote cluster's region). For example, `us-east-1`. - For example, if the origin cluster stores data in a bucket called `my-bucket` in `us-east-1`: [,bash] From 9499847c71b2b6deb6bcefdbb98c39d76708dac6 Mon Sep 17 00:00:00 2001 From: micheleRP Date: Tue, 3 Mar 2026 18:01:21 -0700 Subject: [PATCH 6/9] incorporate Matt's feedback --- modules/manage/partials/remote-read-replicas.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/manage/partials/remote-read-replicas.adoc b/modules/manage/partials/remote-read-replicas.adoc index ead585dda3..ef53823516 100644 --- a/modules/manage/partials/remote-read-replicas.adoc +++ b/modules/manage/partials/remote-read-replicas.adoc @@ -452,7 +452,7 @@ To create a cross-region Remote Read Replica topic, append `region` and `endpoin In the following example, replace the placeholders: -- ``: The name of the original topic on the origin cluster. +- ``: The name of the topic in the cluster hosting the Remote Read Replica. - ``: The S3 bucket configured on the origin cluster (`cloud_storage_bucket`). - ``: The AWS region of the origin cluster's S3 bucket (not the remote cluster's region). From 62a2d492be2cf576d0a6f7856b1a5d214170eca0 Mon Sep 17 00:00:00 2001 From: micheleRP Date: Thu, 5 Mar 2026 11:29:08 -0700 Subject: [PATCH 7/9] Replaced "upstream" with "connection" --- modules/manage/partials/remote-read-replicas.adoc | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/modules/manage/partials/remote-read-replicas.adoc b/modules/manage/partials/remote-read-replicas.adoc index ef53823516..3f9292fc2e 100644 --- a/modules/manage/partials/remote-read-replicas.adoc +++ b/modules/manage/partials/remote-read-replicas.adoc @@ -474,12 +474,12 @@ NOTE: The `endpoint` value must not include the bucket name. Redpanda automatica ==== Limits -Each unique combination of region and endpoint creates a dynamic object storage upstream on the remote cluster. A cluster supports a maximum of 10 active dynamic upstreams. +Each unique combination of region and endpoint creates a separate object storage connection on the remote cluster. A cluster supports a maximum of 10 connections. -How upstreams are counted depends on `cloud_storage_url_style`: +How connections are counted depends on `cloud_storage_url_style`: -- `virtual_host`: Each unique combination of bucket, region, and endpoint counts as one upstream. You can create up to 10 distinct cross-region Remote Read Replica topics per cluster. -- `path`: Each unique combination of region and endpoint counts as one upstream (the bucket name is not part of the key). You can create cross-region Remote Read Replica topics for an unlimited number of buckets, as long as those buckets are spread across no more than 10 distinct region/endpoint combinations. +- `virtual_host`: Each unique combination of bucket, region, and endpoint counts as one connection. You can create up to 10 distinct cross-region Remote Read Replica topics per cluster. +- `path`: Each unique combination of region and endpoint counts as one connection (the bucket name is not part of the key). You can create cross-region Remote Read Replica topics for an unlimited number of buckets, as long as those buckets are spread across no more than 10 distinct region/endpoint combinations. == Reduce lag in data availability From 060a8be6f029c3d649ced1ed244f172375a2c59b Mon Sep 17 00:00:00 2001 From: micheleRP Date: Thu, 5 Mar 2026 13:17:05 -0700 Subject: [PATCH 8/9] Replaced "connection" with "target" --- modules/manage/partials/remote-read-replicas.adoc | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/modules/manage/partials/remote-read-replicas.adoc b/modules/manage/partials/remote-read-replicas.adoc index 3f9292fc2e..3336a9104f 100644 --- a/modules/manage/partials/remote-read-replicas.adoc +++ b/modules/manage/partials/remote-read-replicas.adoc @@ -474,12 +474,12 @@ NOTE: The `endpoint` value must not include the bucket name. Redpanda automatica ==== Limits -Each unique combination of region and endpoint creates a separate object storage connection on the remote cluster. A cluster supports a maximum of 10 connections. +Each unique combination of region and endpoint creates a separate object storage target on the remote cluster. A cluster supports a maximum of 10 targets. -How connections are counted depends on `cloud_storage_url_style`: +How targets are counted depends on `cloud_storage_url_style`: -- `virtual_host`: Each unique combination of bucket, region, and endpoint counts as one connection. You can create up to 10 distinct cross-region Remote Read Replica topics per cluster. -- `path`: Each unique combination of region and endpoint counts as one connection (the bucket name is not part of the key). You can create cross-region Remote Read Replica topics for an unlimited number of buckets, as long as those buckets are spread across no more than 10 distinct region/endpoint combinations. +- `virtual_host`: Each unique combination of bucket, region, and endpoint counts as one target. You can create up to 10 distinct cross-region Remote Read Replica topics for each cluster. +- `path`: Each unique combination of region and endpoint counts as one target (the bucket name is not part of the key). You can create cross-region Remote Read Replica topics for an unlimited number of buckets, as long as those buckets are spread across no more than 10 distinct region/endpoint combinations. == Reduce lag in data availability From 5888fe162777bdd254f764786c524e5f17a324b6 Mon Sep 17 00:00:00 2001 From: micheleRP Date: Thu, 5 Mar 2026 13:22:08 -0700 Subject: [PATCH 9/9] coderabbit suggestions --- modules/manage/partials/remote-read-replicas.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/manage/partials/remote-read-replicas.adoc b/modules/manage/partials/remote-read-replicas.adoc index 3336a9104f..a095bc420d 100644 --- a/modules/manage/partials/remote-read-replicas.adoc +++ b/modules/manage/partials/remote-read-replicas.adoc @@ -470,7 +470,7 @@ rpk topic create my-topic \ -c redpanda.remote.readreplica=my-bucket?region=us-east-1&endpoint=s3.us-east-1.amazonaws.com ---- -NOTE: The `endpoint` value must not include the bucket name. Redpanda automatically prepends the bucket name when using `virtual_host` URL style. +NOTE: The `endpoint` value must not include the bucket name. When using `virtual_host` URL style, Redpanda automatically prepends the bucket name to the endpoint. When using `path` URL style, Redpanda appends the bucket name as a path segment. ==== Limits @@ -479,7 +479,7 @@ Each unique combination of region and endpoint creates a separate object storage How targets are counted depends on `cloud_storage_url_style`: - `virtual_host`: Each unique combination of bucket, region, and endpoint counts as one target. You can create up to 10 distinct cross-region Remote Read Replica topics for each cluster. -- `path`: Each unique combination of region and endpoint counts as one target (the bucket name is not part of the key). You can create cross-region Remote Read Replica topics for an unlimited number of buckets, as long as those buckets are spread across no more than 10 distinct region/endpoint combinations. +- `path`: Each unique combination of region and endpoint counts as one target (the bucket name is not part of the key). You can create cross-region Remote Read Replica topics for multiple buckets using the same region/endpoint combination, with a maximum of 10 distinct region/endpoint combinations for each cluster. == Reduce lag in data availability