Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 47 additions & 3 deletions modules/manage/partials/remote-read-replicas.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,9 @@ You need the following:
* An origin cluster with xref:{tiered-storage-link}#set-up-tiered-storage[Tiered Storage] set up. Multi-region buckets or containers are not supported.
* A topic on the origin cluster, which you can use as a Remote Read Replica topic on the remote cluster.
* A separate remote cluster.
** AWS: The remote cluster must be in the same region as the origin cluster's storage bucket/container.
** AWS: The remote cluster can be in the same or a different region as the origin cluster's S3 bucket. For cross-region Remote Read Replica topics, see <<create-cross-region-rrr-topic>>.
** GCP: The remote cluster can be in the same or a different region as the bucket/container.
** Azure: Remote read replicas are not supported.
** Azure: Remote read replicas are not supported.

include::shared:partial$enterprise-license.adoc[]

Expand All @@ -56,7 +56,7 @@ You must configure access to the same object storage as the origin cluster.
ifndef::env-kubernetes[]
To set up a Remote Read Replica topic on a separate remote cluster:

. Create a remote cluster for the Remote Read Replica topic. For AWS, the remote cluster must be in the same region as the origin cluster's storage bucket/container. For GCP, the remote cluster can be in the same or a different region as the bucket/container.
. Create a remote cluster for the Remote Read Replica topic. For GCP, the remote cluster can be in the same or a different region as the bucket/container. For AWS, the remote cluster can be in the same or a different region, but cross-region Remote Read Replica topics require additional configuration. See <<create-cross-region-rrr-topic>>.
. Run `rpk cluster config edit`, and then specify properties specific to your object storage provider (your cluster will require a restart after any changes to these properties):
+

Expand Down Expand Up @@ -437,6 +437,50 @@ rpk topic create <topic_name> -c redpanda.remote.readreplica=<bucket_name>
* Do not use `redpanda.remote.read` or `redpanda.remote.write` with `redpanda.remote.readreplica`. Redpanda ignores the values for remote read and remote write properties on read replica topics.
====

[[create-cross-region-rrr-topic]]
=== Create a cross-region Remote Read Replica topic on AWS

Use this configuration only when the remote cluster is in a *different AWS region* than the origin cluster's S3 bucket. For same-region AWS or GCP deployments, use the standard <<Create a Remote Read Replica topic,topic creation command>>.

==== Prerequisites

The xref:reference:properties/object-storage-properties.adoc#cloud_storage_url_style[`cloud_storage_url_style`] cluster property must be set explicitly to `virtual_host` or `path` on the remote cluster. The default value does not support cross-region Remote Read Replicas.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nvartolomei Quite surprised to see this. if both styles work, wonder why we force the user to set this explicitly? Very importantly, I don't think this cluster property is settable in Redpanda Cloud (which is where we mostly care about this feature), because how the underlying S3 urls are is not meant to be fiddled with in cloud. Please follow up with the Cloud team to make sure RP CLoud sets this to the style such that this WILL work. In that case, the Cloud docs could omit this step. Otherwise ,we have an issue for cloud

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mattschumpert
... why we force the user to set this explicitly? ...

To cut weeks of effort on core side.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nvartolomei, @mattschumpert Please confirm if we're going to make cloud_storage_url_style configurable in Redpanda Cloud, or if we can conditionalize this Prereqs section out of Cloud docs


==== Create the topic

To create a cross-region Remote Read Replica topic, append `region` and `endpoint` query-string parameters to the bucket name.

In the following example, replace the placeholders:

- `<topic_name>`: The name of the topic in the cluster hosting the Remote Read Replica.
- `<bucket_name>`: The S3 bucket configured on the origin cluster (`cloud_storage_bucket`).
- `<origin_bucket_region>`: The AWS region of the origin cluster's S3 bucket (not the remote cluster's region).

[,bash]
----
rpk topic create <topic_name> \
-c redpanda.remote.readreplica=<bucket_name>?region=<origin_bucket_region>&endpoint=s3.<origin_bucket_region>.amazonaws.com
----

For example, if the origin cluster stores data in a bucket called `my-bucket` in `us-east-1`:

[,bash]
----
rpk topic create my-topic \
-c redpanda.remote.readreplica=my-bucket?region=us-east-1&endpoint=s3.us-east-1.amazonaws.com
----

NOTE: The `endpoint` value must not include the bucket name. When using `virtual_host` URL style, Redpanda automatically prepends the bucket name to the endpoint. When using `path` URL style, Redpanda appends the bucket name as a path segment.

==== Limits

Each unique combination of region and endpoint creates a separate object storage target on the remote cluster. A cluster supports a maximum of 10 targets.

How targets are counted depends on `cloud_storage_url_style`:

- `virtual_host`: Each unique combination of bucket, region, and endpoint counts as one target. You can create up to 10 distinct cross-region Remote Read Replica topics for each cluster.
- `path`: Each unique combination of region and endpoint counts as one target (the bucket name is not part of the key). You can create cross-region Remote Read Replica topics for multiple buckets using the same region/endpoint combination, with a maximum of 10 distinct region/endpoint combinations for each cluster.

== Reduce lag in data availability

:config-ref: cloud_storage_segment_max_upload_interval_sec
Expand Down