From 928c9009eb6239dcbd18738e8cd4b8aa27830be8 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 18 Nov 2025 19:39:47 -0500 Subject: [PATCH 01/75] DOC-1602: Cloud Topics --- modules/ROOT/nav.adoc | 1 + 1 file changed, 1 insertion(+) diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 5eb34695dc..116401cb09 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -37,6 +37,7 @@ ** xref:develop:benchmark.adoc[] ** xref:develop:http-proxy.adoc[] ** xref:develop:config-topics.adoc[] +*** xref:develop:config-topics/cloud-topics.adoc[Cloud Topics] ** xref:console:ui/edit-topic-configuration.adoc[Edit Topic Configuration] ** xref:develop:produce-data/index.adoc[Produce Data] *** xref:develop:produce-data/configure-producers.adoc[] From 2259e38800888c4af7f58bf84f7f826287aebdc9 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 18 Nov 2025 19:47:23 -0500 Subject: [PATCH 02/75] DOC-1602 --- modules/develop/pages/cloud-topics.adoc | 65 +++++++++++++++++++++++++ 1 file changed, 65 insertions(+) create mode 100644 modules/develop/pages/cloud-topics.adoc diff --git a/modules/develop/pages/cloud-topics.adoc b/modules/develop/pages/cloud-topics.adoc new file mode 100644 index 0000000000..f3e735d726 --- /dev/null +++ b/modules/develop/pages/cloud-topics.adoc @@ -0,0 +1,65 @@ += Cloud Topics +:description: Cloud Topics are Kafka topics that enable users to trade off latency for lower costs. +:page-beta: true +:page-categories: Clients, Development +// tag::single-source[] + +ifndef::env-cloud[] +[NOTE] +==== +include::shared:partial$enterprise-license.adoc[] +==== +endif::[] + +While standard Kafka glossterm:topics[] are ideal for high-throughput, cost-sensitive workloads, Cloud Topics are optimized for high-throughput, cost-sensitive workloads. Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud object storage (S3, GCS, MinIO) as the primary mechanism for backing up data. The difference in response time is typically 500ms, which is only impactful for latency-sensitive applications. So when using Cloud Topics, you get the same durability that you have with standard topics, but you also eliminate the significant cross-AZ costs typically incurred when you have high-throughput workloads (for example, for audit logs). + +== Prerequisites + +- rpk v25.3.1: See xref:get-started:rpk-install.adoc[]. +ifndef::env-cloud[] +- xref:manage:tiered-storage.adoc#set-up-tiered-storage[Set up] and xref:manage:tiered-storage.adoc#enable-tiered-storage[enable Tiered Storage] on your Redpanda cluster. +- xref:manage:tiered-storage.adoc#configure-object-storage[Configure object storage] +- include::shared:partial$enterprise-license.adoc[] + +To check if you already have a license key applied to your cluster: + +[,bash] +---- +rpk cluster license info +---- +endif::[] + +== Limitations + +Not all Redpanda features are available when using Cloud Topics (beta). The following features are not supported in this beta release: + +- Kafka compaction +- Iceberg topics +- Topic recovery +- Remote Read Replicas + +IMPORTANT: You should expect to throw away any cluster running the Cloud Topics (beta) after it becomes generally available (GA). Redpanda will be unable to continue using data that was generated by the beta version of this feature. Also, any topic specified to be Cloud Topic cannot subsequently be converted back to a standard, or non-Cloud Topic. + +== Create Cloud Topics + +To create a Cloud Topic: + +[,bash] +---- +rpk topic create -c redpanda.cloud_topic.enable=true +---- ++ +[,bash,role=no-copy] +---- +TOPIC STATUS +audit.analytics.may2025 OK +---- + +To enable a Cloud Topic in a cluster: + +[,bash] +---- +rpk topic create -c redpanda.cloud_topic.enabled=true +---- + +// end::single-source[] \ No newline at end of file From 08f5094ff50d7f80c4e597222f3e959db8b9c9a3 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 18 Nov 2025 20:18:15 -0500 Subject: [PATCH 03/75] DOC-1602: Cloud Topics --- modules/ROOT/nav.adoc | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 116401cb09..c833e5b2f1 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -36,8 +36,9 @@ ** xref:develop:kafka-clients.adoc[Kafka Compatibility] ** xref:develop:benchmark.adoc[] ** xref:develop:http-proxy.adoc[] -** xref:develop:config-topics.adoc[] -*** xref:develop:config-topics/cloud-topics.adoc[Cloud Topics] +** xref:develop:manage-topics/index.adoc[] +*** xref:develop:manage-topics/config-topics.adoc[] +*** xref:develop:manage-topics/cloud-topics.adoc[] ** xref:console:ui/edit-topic-configuration.adoc[Edit Topic Configuration] ** xref:develop:produce-data/index.adoc[Produce Data] *** xref:develop:produce-data/configure-producers.adoc[] From d1e0dd11f00417752269f235453283ef299c9b11 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 18 Nov 2025 20:18:36 -0500 Subject: [PATCH 04/75] DOC-1602: Cloud Topics --- .../deploy/pages/redpanda/manual/production/dev-deployment.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/deploy/pages/redpanda/manual/production/dev-deployment.adoc b/modules/deploy/pages/redpanda/manual/production/dev-deployment.adoc index 36e5db81e1..44aaf8cd91 100644 --- a/modules/deploy/pages/redpanda/manual/production/dev-deployment.adoc +++ b/modules/deploy/pages/redpanda/manual/production/dev-deployment.adoc @@ -8,7 +8,7 @@ You can deploy Redpanda using well-known configuration properties optimized for [NOTE] ==== -* Development mode enables write caching by default. This is a relaxed mode of xref:develop:produce-data/configure-producers.adoc#acksall[`acks=all`] that acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to fsync to disk. Write caching provides lower latency while still ensuring that a majority of brokers acknowledge the write. For more information, or to disable this, see xref:develop:config-topics.adoc#configure-write-caching[write caching]. +* Development mode enables write caching by default. This is a relaxed mode of xref:develop:produce-data/configure-producers.adoc#acksall[`acks=all`] that acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to fsync to disk. Write caching provides lower latency while still ensuring that a majority of brokers acknowledge the write. For more information, or to disable this, see xref:develop:manage-topics/config-topics.adoc#configure-write-caching[write caching]. * Development mode also bypasses `fsync`, acknowledging messages before they're stored to disk. This reduces the durability of messages, could cause potential data loss, and could give unrealistic performance characteristics for a production environment. ==== From 89bae92ad8b66f4b50f7392a77cccda998983bf5 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 18 Nov 2025 20:19:02 -0500 Subject: [PATCH 05/75] DOC-1602: Cloud Topics --- modules/console/pages/ui/edit-topic-configuration.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/console/pages/ui/edit-topic-configuration.adoc b/modules/console/pages/ui/edit-topic-configuration.adoc index 75dcd24042..ab9207a367 100644 --- a/modules/console/pages/ui/edit-topic-configuration.adoc +++ b/modules/console/pages/ui/edit-topic-configuration.adoc @@ -18,4 +18,4 @@ == Suggested reading - xref:reference:properties/topic-properties.adoc[] -- xref:develop:config-topics.adoc[] \ No newline at end of file +- xref:develop:manage-topics/config-topics.adoc[] \ No newline at end of file From 5a71ac24e1e5773c6bb1db55316b8b05f866096e Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 18 Nov 2025 20:19:16 -0500 Subject: [PATCH 06/75] DOC-1602 --- modules/manage/partials/audit-logging.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/manage/partials/audit-logging.adoc b/modules/manage/partials/audit-logging.adoc index ad1fe36a02..faa1714f8e 100644 --- a/modules/manage/partials/audit-logging.adoc +++ b/modules/manage/partials/audit-logging.adoc @@ -433,6 +433,6 @@ xref:manage:audit-logging/audit-log-samples.adoc[See samples of audit log messag include::shared:partial$suggested-reading.adoc[] - xref:reference:topic-properties.adoc[] -- xref:develop:config-topics.adoc[] +- xref:develop:manage-topics/config-topics.adoc[] endif::[] From d5612432b61ba25c9b9db2acef9c96efbc94a1a2 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 18 Nov 2025 20:19:29 -0500 Subject: [PATCH 07/75] DOC-1602 --- modules/reference/pages/properties/cluster-properties.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/reference/pages/properties/cluster-properties.adoc b/modules/reference/pages/properties/cluster-properties.adoc index f3460c1adc..053173d11d 100644 --- a/modules/reference/pages/properties/cluster-properties.adoc +++ b/modules/reference/pages/properties/cluster-properties.adoc @@ -10,4 +10,4 @@ NOTE: Some cluster properties require that you restart the cluster for any updat == Cluster configuration -include::reference:partial$properties/cluster-properties.adoc[tags=!deprecated;!exclude-from-docs] \ No newline at end of file +include::reference:partial$properties/cluster-properties.adoc[tags=!deprecated;!exclude-from-docs] From 79fcaf48e28b7fd74568c80b680016de93a79b2c Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 18 Nov 2025 20:19:42 -0500 Subject: [PATCH 08/75] DOC-1602 --- .../pages/properties/topic-properties.adoc | 106 ++++++++++++++++++ 1 file changed, 106 insertions(+) diff --git a/modules/reference/pages/properties/topic-properties.adoc b/modules/reference/pages/properties/topic-properties.adoc index ce669ba8e9..e99bff7283 100644 --- a/modules/reference/pages/properties/topic-properties.adoc +++ b/modules/reference/pages/properties/topic-properties.adoc @@ -33,6 +33,112 @@ These properties control disk flushing, replication, and how topics interact wit include::reference:partial$properties/topic-properties.adoc[tags=category-performance-cluster;!deprecated;!exclude-from-docs] +The maximum bytes not fsynced per partition. If this configured threshold is reached, the log is automatically fsynced, even though it wasn't explicitly requested. + +*Type:* integer + +*Unit:* bytes + +*Accepted values:* [`1`, `9223372036854775807`] + +*Default:* `262144` + +*Related cluster property:* xref:./cluster-properties.adoc#flush_bytes[`flush_bytes`] + +**Related topics**: + +- xref:develop:produce-data/configure-producers.adoc[] + +--- +[[flushms]] +=== flush.ms + +The maximum delay (in ms) between two subsequent fsyncs. After this delay, the log is automatically fsynced. + +*Type:* integer + +*Unit:* milliseconds + +*Accepted values:* [`1`, `9223372036854775`] + +*Default:* `100` + +*Related cluster property:* xref:./cluster-properties.adoc#flush_ms[`flush_ms`] + +**Related topics**: + +- xref:develop:produce-data/configure-producers.adoc[] + +--- +[[redpandaleaderspreference]] +=== redpanda.leaders.preference + +The preferred location (rack) for partition leaders of a topic. + +This property inherits the value from the config_ref:default_leaders_preference,true,properties/cluster-properties[] cluster configuration property. You may override the cluster-wide setting by specifying the value for individual topics. + +If the cluster configuration property config_ref:enable_rack_awareness,true,properties/cluster-properties[] is set to `false`, Leader Pinning is disabled across the cluster. + +*Type:* string + +*Default:* `none` + +**Values**: + +- `none`: Opt out the topic from Leader Pinning. +- `racks:[,,...]`: Specify the preferred location (rack) of all topic partition leaders. The list can contain one or more rack IDs. If you specify multiple IDs, Redpanda tries to distribute the partition leader locations equally across brokers in these racks. + +**Related topics**: + +- xref:develop:produce-data/leader-pinning.adoc[Leader pinning] + +--- +[[replicationfactor]] +=== replication.factor + +The number of replicas of a topic to save in different nodes (brokers) of a cluster. + +If `replication.factor` is set to a positive value, it overrides the cluster property xref:./cluster-properties.adoc#default_topic_replication[default_topic_replication] for the topic. + +NOTE: Although `replication.factor` isn't returned or displayed by xref:reference:rpk/rpk-topic/rpk-topic-describe.adoc[`rpk topic describe`] as a valid Kafka property, you can set it using xref:reference:rpk/rpk-topic/rpk-topic-alter-config.adoc[`rpk topic alter-config`]. When the `replication.factor` of a topic is altered, it isn't simply a property value that's updated, but rather the actual replica sets of topic partitions that are changed. + +*Type:* integer + +*Accepted values:* [`1`, `512`] + +*Default:* null + +*Related cluster property:* xref:./cluster-properties.adoc#default_topic_replication[`default_topic_replication`] + +**Related topics**: + +- xref:develop:manage-topics/config-topics.adoc#choose-the-replication-factor[Choose the replication factor] +- xref:develop:manage-topics/config-topics.adoc#change-the-replication-factor[Change the replication factor] + +--- +[[writecaching]] +=== write.caching + +The write caching mode to apply to a topic. + +When `write.caching` is set, it overrides the cluster property xref:cluster-properties.adoc#write_caching_default[`write_caching_default`]. Write caching acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to be written to disk. With `acks=all`, this provides lower latency while still ensuring that a majority of brokers acknowledge the write. Fsyncs follow <> and <>, whichever is reached first. + +*Type:* boolean + +*Default:* `false` + +**Values**: + +- `true` - Enables write caching for a topic, according to <> and <>. +- `false` - Disables write caching for a topic, according to <> and <>. + +*Related cluster property:* xref:./cluster-properties.adoc#write_caching_default[`write_caching_default`] + +**Related topics**: + +- xref:develop:manage-topics/config-topics.adoc#configure-write-caching[Write caching] + +--- == Tiered Storage properties Configure properties to manage topics for xref:manage:tiered-storage.adoc[Tiered Storage]. From 471306a2b844474faf497251f095199b3b51a91c Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 18 Nov 2025 20:19:55 -0500 Subject: [PATCH 09/75] DOC-1602 --- modules/reference/pages/rpk/rpk-redpanda/rpk-redpanda-mode.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/reference/pages/rpk/rpk-redpanda/rpk-redpanda-mode.adoc b/modules/reference/pages/rpk/rpk-redpanda/rpk-redpanda-mode.adoc index 9cabe78320..efd6b2a553 100644 --- a/modules/reference/pages/rpk/rpk-redpanda/rpk-redpanda-mode.adoc +++ b/modules/reference/pages/rpk/rpk-redpanda/rpk-redpanda-mode.adoc @@ -10,7 +10,7 @@ Development mode (`development` or `dev`) includes the following development-onl * Sets `developer_mode` to `true`. This starts Redpanda with dev-mode only settings, including: ** No minimal memory limits are enforced. ** No core assignment rules for Redpanda nodes are enforced. - ** Enables write caching, which is a relaxed mode of `acks=all` that acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to fsync to disk. This provides lower latency while still ensuring that a majority of brokers acknowledge the write. For more information, or to disable this, see xref:develop:config-topics.adoc#configure-write-caching[write caching]. + ** Enables write caching, which is a relaxed mode of `acks=all` that acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to fsync to disk. This provides lower latency while still ensuring that a majority of brokers acknowledge the write. For more information, or to disable this, see xref:develop:manage-topics/config-topics.adoc#configure-write-caching[write caching]. ** Bypasses `fsync` (from https://docs.seastar.io/master/structseastar_1_1reactor%5F%5Foptions.html#ad66cb23f59ed5dfa8be8189313988692[Seastar option `unsafe_bypass_fsync`^]), which results in unrealistically fast clusters and may result in data loss. * Sets `overprovisioned` to `true`. Redpanda expects a dev system to be an overprovisioned environment. Based on a https://docs.seastar.io/master/structseastar_1_1reactor%5F%5Foptions.html#a0caf6c2ad579b8c22e1352d796ec3c1d[Seastar option^], setting `overprovisioned` disables thread affinity, zeros idle polling time, and disables busy-poll for disk I/O. * Sets all autotuner xref:./rpk-redpanda-tune-list.adoc#tuners[tuners] to `false`. The tuners are intended to run only for production mode. From 2e552eee7df3e9285765a7d0ceab9f7249cf8343 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 18 Nov 2025 20:20:25 -0500 Subject: [PATCH 10/75] DOC-1602 --- .../pages/cluster-maintenance/topic-property-configuration.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc b/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc index 8a182e138a..7f1d6410f1 100644 --- a/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc +++ b/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc @@ -220,6 +220,6 @@ For complete details about all available topic properties, see xref:reference:pr * xref:reference:properties/topic-properties.adoc[Topic Configuration Properties] - Complete reference of all available topic properties * xref:manage:cluster-maintenance/cluster-property-configuration.adoc[Configure Cluster Properties] - Configure cluster-wide defaults -* xref:develop:config-topics.adoc[Manage Topics] - Create and manage topics +* xref:develop:manage-topics/config-topics.adoc[Manage Topics] - Create and manage topics * xref:manage:kubernetes/k-manage-topics.adoc[Manage Topics in Kubernetes] - Topic management in Kubernetes deployments * xref:console:ui/edit-topic-configuration.adoc[Edit Topic Configuration in Redpanda Console] - Graphical topic configuration \ No newline at end of file From 47f06ec0a7ffb0365e869d6b47041c916b291ee3 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 18 Nov 2025 20:20:37 -0500 Subject: [PATCH 11/75] DOC-1602 --- modules/manage/pages/cluster-maintenance/disk-utilization.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/manage/pages/cluster-maintenance/disk-utilization.adoc b/modules/manage/pages/cluster-maintenance/disk-utilization.adoc index 8d66849891..b1484a49a7 100644 --- a/modules/manage/pages/cluster-maintenance/disk-utilization.adoc +++ b/modules/manage/pages/cluster-maintenance/disk-utilization.adoc @@ -85,7 +85,7 @@ Redpanda runs a log cleanup process in the background to apply these policy sett See also: * xref:manage:tiered-storage.adoc#manage-local-capacity-for-tiered-storage-topics[Manage local capacity for Tiered Storage topics] -* xref:develop:config-topics.adoc#delete-records-from-a-topic[Delete records from a topic] +* xref:develop:manage-topics/config-topics.adoc#delete-records-from-a-topic[Delete records from a topic] [[set-time-based-retention]] === Set time-based retention From 2bd2b3863d2b300ef8522550234c01677a9e1418 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 18 Nov 2025 20:20:50 -0500 Subject: [PATCH 12/75] DOC-1602 --- .../manage/pages/cluster-maintenance/compaction-settings.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/manage/pages/cluster-maintenance/compaction-settings.adoc b/modules/manage/pages/cluster-maintenance/compaction-settings.adoc index a4c55a2f79..e0d528b982 100644 --- a/modules/manage/pages/cluster-maintenance/compaction-settings.adoc +++ b/modules/manage/pages/cluster-maintenance/compaction-settings.adoc @@ -12,7 +12,7 @@ image::shared:compaction-example.png[Example of topic compaction] This diagram illustrates a compacted topic. Imagine a remote sensor network that uses image recognition to track appearances of red pandas in a geographic area. The sensor network employs special devices that send records to a topic when they detect one. You might enable compaction to reduce topic storage while still maintaining a record in the topic of the last time each device saw a red panda, perhaps to see if they stop frequenting a given area. The left side of the diagram shows all records sent across the topic. The right side illustrates the results of compaction; older records for certain keys are deleted from the log. -NOTE: If your application requires consuming every record for a given key, consider using the `delete` xref:develop:config-topics#change-the-cleanup-policy.adoc[cleanup policy] instead. +NOTE: If your application requires consuming every record for a given key, consider using the `delete` xref:develop:manage-topics/config-topics.adoc#change-the-cleanup-policy[cleanup policy] instead. IMPORTANT: When using xref:manage:tiered-storage.adoc[Tiered Storage], compaction functions at the local storage level. As long as a segment remains in local storage, its records are eligible for compaction. Once a segment is uploaded to object storage and removed from local storage it is not retrieved for further compaction operations. A key may therefore appear in multiple segments between Tiered Storage and local storage. From c1c78fb835258511b53f6ef25b4b87ae3d302123 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 18 Nov 2025 20:21:06 -0500 Subject: [PATCH 13/75] DOC-1602 --- modules/get-started/pages/quick-start.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/get-started/pages/quick-start.adoc b/modules/get-started/pages/quick-start.adoc index 1c65c751f1..ebbab608be 100644 --- a/modules/get-started/pages/quick-start.adoc +++ b/modules/get-started/pages/quick-start.adoc @@ -443,7 +443,7 @@ docker exec -it redpanda-0 rpk topic create chat-room \ -X pass=secretpassword ``` + -<1> Set a replication factor of 3 to replicate the topic across all 3 brokers. This replication factor provides high availability and data durability. For more details, see xref:develop:config-topics.adoc#choose-the-replication-factor[Choose the replication factor]. +<1> Set a replication factor of 3 to replicate the topic across all 3 brokers. This replication factor provides high availability and data durability. For more details, see xref:develop:manage-topics/config-topics.adoc#choose-the-replication-factor[Choose the replication factor]. <2> Enable remote reads for this topic to read offloaded records from object storage. <3> Enable remote writes for this topic to offload older records to object storage. For more details, see xref:manage:tiered-storage.adoc[]. + From 2811fe066c164e61f1b60fe1f555fcc777ce2254 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 18 Nov 2025 20:21:25 -0500 Subject: [PATCH 14/75] DOC-1602 --- modules/develop/pages/kafka-clients.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/develop/pages/kafka-clients.adoc b/modules/develop/pages/kafka-clients.adoc index 3f7815ded2..f984c9db93 100644 --- a/modules/develop/pages/kafka-clients.adoc +++ b/modules/develop/pages/kafka-clients.adoc @@ -60,7 +60,7 @@ endif::[] * HTTP Proxy (pandaproxy): Unlike other REST proxy implementations in the Kafka ecosystem, Redpanda HTTP Proxy does not support topic and ACLs CRUD through the HTTP Proxy. HTTP Proxy is designed for clients producing and consuming data that do not perform administrative functions. ifdef::env-cloud[] + -* The `delete.retention.ms` topic configuration in Kafka is not supported. Tombstone markers are not removed for topics with a `compact` xref:get-started:config-topics.adoc#change-the-cleanup-policy[cleanup policy]. Redpanda only deletes tombstone markers when topics with a cleanup policy of `compact,delete` have reached their xref:get-started:create-topic.adoc[retention limits]. +* The `delete.retention.ms` topic configuration in Kafka is not supported. Tombstone markers are not removed for topics with a `compact` xref:develop:manage-topics/config-topics.adoc#change-the-cleanup-policy[cleanup policy]. Redpanda only deletes tombstone markers when topics with a cleanup policy of `compact,delete` have reached their retention limits. endif::[] ifndef::env-cloud[] + From db75f1f7cead5df1753f4cc0882661dc043d69fa Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 18 Nov 2025 20:21:48 -0500 Subject: [PATCH 15/75] DOC-1602 --- modules/develop/pages/cloud-topics.adoc | 65 ------------------------- 1 file changed, 65 deletions(-) delete mode 100644 modules/develop/pages/cloud-topics.adoc diff --git a/modules/develop/pages/cloud-topics.adoc b/modules/develop/pages/cloud-topics.adoc deleted file mode 100644 index f3e735d726..0000000000 --- a/modules/develop/pages/cloud-topics.adoc +++ /dev/null @@ -1,65 +0,0 @@ -= Cloud Topics -:description: Cloud Topics are Kafka topics that enable users to trade off latency for lower costs. -:page-beta: true -:page-categories: Clients, Development -// tag::single-source[] - -ifndef::env-cloud[] -[NOTE] -==== -include::shared:partial$enterprise-license.adoc[] -==== -endif::[] - -While standard Kafka glossterm:topics[] are ideal for high-throughput, cost-sensitive workloads, Cloud Topics are optimized for high-throughput, cost-sensitive workloads. Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud object storage (S3, GCS, MinIO) as the primary mechanism for backing up data. The difference in response time is typically 500ms, which is only impactful for latency-sensitive applications. So when using Cloud Topics, you get the same durability that you have with standard topics, but you also eliminate the significant cross-AZ costs typically incurred when you have high-throughput workloads (for example, for audit logs). - -== Prerequisites - -- rpk v25.3.1: See xref:get-started:rpk-install.adoc[]. -ifndef::env-cloud[] -- xref:manage:tiered-storage.adoc#set-up-tiered-storage[Set up] and xref:manage:tiered-storage.adoc#enable-tiered-storage[enable Tiered Storage] on your Redpanda cluster. -- xref:manage:tiered-storage.adoc#configure-object-storage[Configure object storage] -- include::shared:partial$enterprise-license.adoc[] - -To check if you already have a license key applied to your cluster: - -[,bash] ----- -rpk cluster license info ----- -endif::[] - -== Limitations - -Not all Redpanda features are available when using Cloud Topics (beta). The following features are not supported in this beta release: - -- Kafka compaction -- Iceberg topics -- Topic recovery -- Remote Read Replicas - -IMPORTANT: You should expect to throw away any cluster running the Cloud Topics (beta) after it becomes generally available (GA). Redpanda will be unable to continue using data that was generated by the beta version of this feature. Also, any topic specified to be Cloud Topic cannot subsequently be converted back to a standard, or non-Cloud Topic. - -== Create Cloud Topics - -To create a Cloud Topic: - -[,bash] ----- -rpk topic create -c redpanda.cloud_topic.enable=true ----- -+ -[,bash,role=no-copy] ----- -TOPIC STATUS -audit.analytics.may2025 OK ----- - -To enable a Cloud Topic in a cluster: - -[,bash] ----- -rpk topic create -c redpanda.cloud_topic.enabled=true ----- - -// end::single-source[] \ No newline at end of file From bf1a074fe140814c59b556235d0874f7ad0d6131 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 18 Nov 2025 20:22:03 -0500 Subject: [PATCH 16/75] DOC-1602 --- modules/develop/pages/produce-data/configure-producers.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/develop/pages/produce-data/configure-producers.adoc b/modules/develop/pages/produce-data/configure-producers.adoc index 18de11b961..77a9bc4a3a 100644 --- a/modules/develop/pages/produce-data/configure-producers.adoc +++ b/modules/develop/pages/produce-data/configure-producers.adoc @@ -74,7 +74,7 @@ Kafka, a message is considered acknowledged without the requirement that it has been fsynced. Messages that have not been fsynced to disk may be lost in the event of a broker crash. So when using `acks=all`, the Redpanda default configuration is more resilient than Kafka's. You can also consider -using xref:develop:config-topics.adoc#configure-write-caching[write caching], which is a relaxed mode of `acks=all` that acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to fsync to disk. This provides lower latency while still ensuring that a majority of brokers acknowledge the write. +using xref:develop:manage-topics/config-topics.adoc#configure-write-caching[write caching], which is a relaxed mode of `acks=all` that acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to fsync to disk. This provides lower latency while still ensuring that a majority of brokers acknowledge the write. endif::[] From 6b23790d64bd1e39d9dc6a3fd6673fb18e9b5e8a Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 18 Nov 2025 20:22:14 -0500 Subject: [PATCH 17/75] DOC-1602 --- modules/develop/pages/manage-topics/index.adoc | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 modules/develop/pages/manage-topics/index.adoc diff --git a/modules/develop/pages/manage-topics/index.adoc b/modules/develop/pages/manage-topics/index.adoc new file mode 100644 index 0000000000..37897f4414 --- /dev/null +++ b/modules/develop/pages/manage-topics/index.adoc @@ -0,0 +1,4 @@ += Topics +:page-categories: Clients, Development +:description: Learn how to manage topics in Redpanda, including creation, configuration, and advanced features. +:page-layout: index \ No newline at end of file From d34750b63925192eaeee53411cc54cd420fe2561 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 18 Nov 2025 20:22:30 -0500 Subject: [PATCH 18/75] DOC-1602 --- .../pages/manage-topics/config-topics.adoc | 276 ++++++++++++++++++ 1 file changed, 276 insertions(+) create mode 100644 modules/develop/pages/manage-topics/config-topics.adoc diff --git a/modules/develop/pages/manage-topics/config-topics.adoc b/modules/develop/pages/manage-topics/config-topics.adoc new file mode 100644 index 0000000000..9c361c641b --- /dev/null +++ b/modules/develop/pages/manage-topics/config-topics.adoc @@ -0,0 +1,276 @@ += Manage Topics +:page-categories: Clients, Development +:description: Learn how to create topics, update topic configurations, and delete topics or records. +:page-aliases: develop:config-topics.adoc +// tag::single-source[] + +include::develop:partial$topic-defaults.adoc[] + +== Create a topic + +Creating a topic can be as simple as specifying a name for your topic on the command line. For example, to create a topic named `xyz`, run: + +[,bash] +---- +rpk topic create xyz +---- + +ifndef::env-cloud[] +This command creates a topic named `xyz` with one partition and one replica, because these are the default values set in the cluster configuration file. Replicas are copies of partitions that are distributed across different brokers, so if one broker goes down, other brokers still have a copy of the data. + +endif::[] + +ifdef::env-cloud[] +This command creates a topic named `xyz` with one partition and three replicas, because these are the default values set in the cluster configuration file. Replicas are copies of partitions that are distributed across different brokers, so if one broker goes down, other brokers still have a copy of the data. + +Redpanda Cloud supports 40,000 topics per cluster. + +endif::[] + +=== Choose the number of partitions + +A partition acts as a log file where topic data is written. Dividing topics into partitions allows producers to write messages in parallel and consumers to read messages in parallel. The higher the number of partitions, the greater the throughput. + +TIP: As a general rule, select a number of partitions that corresponds to the maximum number of consumers in any consumer group that will consume the data. + +For example, suppose you plan to create a consumer group with 10 consumers. To create topic `xyz` with 10 partitions, run: + +[,bash] +---- +rpk topic create xyz -p 10 +---- + +ifndef::env-cloud[] +=== Choose the replication factor + +The default replication factor in the cluster configuration is set to 1. By choosing a replication factor greater than 1, you ensure that each partition has a copy of its data on at least one other broker. One replica acts as the leader, and the other replicas are followers. + +To specify a replication factor of 3 for topic `xyz`, run: + +[,bash] +---- +rpk topic create xyz -r 3 +---- + +NOTE: The replication factor must be an odd number. Redpanda Data recommends a replication factor of 3 for most use cases. Administrators may set a minimum required replication factor for any new topic in the cluster through the cluster-level xref:reference:cluster-properties.adoc#minimum_topic_replications[`minimum_topic_replications`] property. + +TIP: If you enable xref:manage:tiered-storage.adoc[Tiered Storage] on a topic, you can then use xref:manage:topic-recovery.adoc[topic recovery] to restore data for a deleted topic. + +endif::[] + +== Update topic configurations + +After you create a topic, you can update the topic property settings for all new data written to it. For example, you can add partitions or change the cleanup policy. + +=== Add partitions + +You can assign a certain number of partitions when you create a topic, and add partitions later. For example, suppose you add brokers to your cluster, and you want to take advantage of the additional processing power. To increase the number of partitions for existing topics, run: + +[,bash] +---- +rpk topic add-partitions [TOPICS...] --num [#] +---- + +Note that `--num <#>` is the number of partitions to _add_, not the total number of partitions. + +include::develop:partial$balance-existing-topic-redistribution.adoc[] + +ifndef::env-cloud[] +=== Change the replication factor + +Suppose you create a topic with the default replication factor of 1 (which is specified in the cluster properties configuration file). Now you want to change the replication factor to 3, so you can have two backups of topic data in case a broker goes down. To set the replication factor to 3, run: + +[,bash] +---- +rpk topic alter-config [TOPICS...] --set replication.factor=3 +---- + +NOTE: The replication factor can't exceed the number of Redpanda brokers. If you try to set a replication factor greater than the number of brokers, the request is rejected. + +endif::[] + + +=== Change the cleanup policy + +The cleanup policy determines how to clean up the partition log files when they reach a certain size: + +* `delete` deletes data based on age or log size. Topics retain all records until then. +* `compact` compacts the data by only keeping the latest values for each KEY. +* `compact,delete` combines both methods. + +Unlike compacted topics, which keep only the most recent message for a given key, topics configured with a `delete` cleanup policy provide a running history of all changes for those topics. + +include::develop:partial$topic-properties-warning.adoc[] + +For example, to change a topic's policy to `compact`, run: + +[,bash] +---- +rpk topic alter-config [TOPICS…] —-set cleanup.policy=compact +---- + +ifndef::env-cloud[] +For details on compaction in Redpanda, see xref:manage:cluster-maintenance/compaction-settings.adoc[Compaction settings]. + +endif::[] + +=== Configure write caching + +Write caching is a relaxed mode of xref:develop:produce-data/configure-producers.adoc#acksall[`acks=all`] that provides better performance at the expense of durability. It acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to be written to disk. This provides lower latency while still ensuring that a majority of brokers acknowledge the write. + +Write caching applies to user topics. It does not apply to transactions or consumer offsets: data written in the context of a transaction and consumer offset commits is always written to disk and fsynced before being acknowledged to the client. + +ifndef::env-cloud[] +NOTE: For clusters in xref:reference:rpk/rpk-redpanda/rpk-redpanda-mode.adoc#development-mode[development mode], write caching is enabled by default. For clusters in production mode, it is disabled by default. + +endif::[] + +Only enable write caching on workloads that can tolerate some data loss in the case of multiple, simultaneous broker failures. Leaving write caching disabled safeguards your data against complete data center or availability zone failures. + +ifndef::env-cloud[] + +==== Configure at cluster level + +To enable write caching by default in all user topics, set the cluster-level property xref:reference:cluster-properties.adoc#write_caching_default[`write_caching_default`]: + +`rpk cluster config set write_caching_default=true` + +With `write_caching_default` set to true at the cluster level, Redpanda fsyncs to disk according to xref:reference:cluster-properties.adoc#raft_replica_max_pending_flush_bytes[`raft_replica_max_pending_flush_bytes`] and xref:reference:cluster-properties.adoc#raft_replica_max_flush_delay_ms[`raft_replica_max_flush_delay_ms`], whichever is reached first. + +endif::[] + +==== Configure at topic level + +To override the cluster-level setting at the topic level, set the topic-level property `write.caching`: + +`rpk topic alter-config my_topic --set write.caching=true` + +With `write.caching` enabled at the topic level, Redpanda fsyncs to disk according to `flush.ms` and `flush.bytes`, whichever is reached first. + +=== Remove a configuration setting + +You can remove a configuration that overrides the default setting, and the setting will use the default value again. For example, suppose you altered the cleanup policy to use `compact` instead of the default, `delete`. Now you want to return the policy setting to the default. To remove the configuration setting `cleanup.policy=compact`, run `rpk topic alter-config` with the `--delete` flag: + +[,bash] +---- +rpk topic alter-config [TOPICS...] --delete cleanup.policy +---- + +== List topic configuration settings + +To display all the configuration settings for a topic, run: + +[,bash] +---- +rpk topic describe -c +---- + +The `-c` flag limits the command output to just the topic configurations. This command is useful for checking the default configuration settings before you make any changes and for verifying changes after you make them. + +The following command output displays after running `rpk topic describe test-topic`, where `test-topic` was created with default settings: + +ifndef::env-cloud[] +[,bash] +---- +rpk topic describe test_topic +SUMMARY +======= +NAME test_topic +PARTITIONS 1 +REPLICAS 1 + +CONFIGS +======= +KEY VALUE SOURCE +cleanup.policy delete DYNAMIC_TOPIC_CONFIG +compression.type producer DEFAULT_CONFIG +max.message.bytes 1048576 DEFAULT_CONFIG +message.timestamp.type CreateTime DEFAULT_CONFIG +redpanda.datapolicy function_name: script_name: DEFAULT_CONFIG +redpanda.remote.delete true DEFAULT_CONFIG +redpanda.remote.read false DEFAULT_CONFIG +redpanda.remote.write false DEFAULT_CONFIG +retention.bytes -1 DEFAULT_CONFIG +retention.local.target.bytes -1 DEFAULT_CONFIG +retention.local.target.ms 86400000 DEFAULT_CONFIG +retention.ms 604800000 DEFAULT_CONFIG +segment.bytes 1073741824 DEFAULT_CONFIG +---- + +Suppose you add two partitions, and increase the number of replicas to 3. The new command output confirms the changes in the `SUMMARY` section: + +[.no-copy] +---- +SUMMARY +======= +NAME test_topic +PARTITIONS 3 +REPLICAS 3 +---- + +endif::[] + +ifdef::env-cloud[] +[,bash] +---- +rpk topic describe test_topic +SUMMARY +======= +NAME test_topic +PARTITIONS 1 +REPLICAS 3 + +CONFIGS +======= +KEY VALUE SOURCE +cleanup.policy delete DYNAMIC_TOPIC_CONFIG +compression.type producer DEFAULT_CONFIG +max.message.bytes 1048576 DEFAULT_CONFIG +message.timestamp.type CreateTime DEFAULT_CONFIG +redpanda.datapolicy function_name: script_name: DEFAULT_CONFIG +redpanda.remote.delete true DEFAULT_CONFIG +redpanda.remote.read false DEFAULT_CONFIG +redpanda.remote.write false DEFAULT_CONFIG +retention.bytes -1 DEFAULT_CONFIG +retention.local.target.bytes -1 DEFAULT_CONFIG +retention.local.target.ms 86400000 DEFAULT_CONFIG +retention.ms 604800000 DEFAULT_CONFIG +segment.bytes 1073741824 DEFAULT_CONFIG +---- + +endif::[] + +== Delete a topic + +To delete a topic, run: + +[,bash] +---- +rpk topic delete +---- + +When a topic is deleted, its underlying data is deleted, too. + +To delete multiple topics at a time, provide a space-separated list. For example, to delete two topics named `topic1` and `topic2`, run: + +[,bash] +---- +rpk topic delete topic1 topic2 +---- + +You can also use the `-r` flag to specify one or more regular expressions; then, any topic names that match the pattern you specify are deleted. For example, to delete topics with names that start with "`f`" and end with "`r`", run: + +[,bash] +---- +rpk topic delete -r '^f.*' '.*r$' +---- + +Note that the first regular expression must start with the `^` symbol, and the last expression must end with the `$` symbol. This requirement helps prevent accidental deletions. + +include::develop:partial$delete-topic-records.adoc[] + +== Next steps + +xref:develop:produce-data/configure-producers.adoc[] + +// end::single-source[] From fcd88ccdc98714a6956e12be6bc4327b0bf129e3 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 18 Nov 2025 20:22:43 -0500 Subject: [PATCH 19/75] DOC-1602 --- .../pages/manage-topics/cloud-topics.adoc | 65 +++++++++++++++++++ 1 file changed, 65 insertions(+) create mode 100644 modules/develop/pages/manage-topics/cloud-topics.adoc diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc new file mode 100644 index 0000000000..f3e735d726 --- /dev/null +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -0,0 +1,65 @@ += Cloud Topics +:description: Cloud Topics are Kafka topics that enable users to trade off latency for lower costs. +:page-beta: true +:page-categories: Clients, Development +// tag::single-source[] + +ifndef::env-cloud[] +[NOTE] +==== +include::shared:partial$enterprise-license.adoc[] +==== +endif::[] + +While standard Kafka glossterm:topics[] are ideal for high-throughput, cost-sensitive workloads, Cloud Topics are optimized for high-throughput, cost-sensitive workloads. Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud object storage (S3, GCS, MinIO) as the primary mechanism for backing up data. The difference in response time is typically 500ms, which is only impactful for latency-sensitive applications. So when using Cloud Topics, you get the same durability that you have with standard topics, but you also eliminate the significant cross-AZ costs typically incurred when you have high-throughput workloads (for example, for audit logs). + +== Prerequisites + +- rpk v25.3.1: See xref:get-started:rpk-install.adoc[]. +ifndef::env-cloud[] +- xref:manage:tiered-storage.adoc#set-up-tiered-storage[Set up] and xref:manage:tiered-storage.adoc#enable-tiered-storage[enable Tiered Storage] on your Redpanda cluster. +- xref:manage:tiered-storage.adoc#configure-object-storage[Configure object storage] +- include::shared:partial$enterprise-license.adoc[] + +To check if you already have a license key applied to your cluster: + +[,bash] +---- +rpk cluster license info +---- +endif::[] + +== Limitations + +Not all Redpanda features are available when using Cloud Topics (beta). The following features are not supported in this beta release: + +- Kafka compaction +- Iceberg topics +- Topic recovery +- Remote Read Replicas + +IMPORTANT: You should expect to throw away any cluster running the Cloud Topics (beta) after it becomes generally available (GA). Redpanda will be unable to continue using data that was generated by the beta version of this feature. Also, any topic specified to be Cloud Topic cannot subsequently be converted back to a standard, or non-Cloud Topic. + +== Create Cloud Topics + +To create a Cloud Topic: + +[,bash] +---- +rpk topic create -c redpanda.cloud_topic.enable=true +---- ++ +[,bash,role=no-copy] +---- +TOPIC STATUS +audit.analytics.may2025 OK +---- + +To enable a Cloud Topic in a cluster: + +[,bash] +---- +rpk topic create -c redpanda.cloud_topic.enabled=true +---- + +// end::single-source[] \ No newline at end of file From b9b2ae455177aa42e38dcb0c8d05035f8869f109 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 18 Nov 2025 20:42:54 -0500 Subject: [PATCH 20/75] WIP --- modules/develop/pages/manage-topics/cloud-topics.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index f3e735d726..a6544ab912 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -38,7 +38,7 @@ Not all Redpanda features are available when using Cloud Topics (beta). The foll - Topic recovery - Remote Read Replicas -IMPORTANT: You should expect to throw away any cluster running the Cloud Topics (beta) after it becomes generally available (GA). Redpanda will be unable to continue using data that was generated by the beta version of this feature. Also, any topic specified to be Cloud Topic cannot subsequently be converted back to a standard, or non-Cloud Topic. +IMPORTANT: You should expect to throw away any cluster running the Cloud Topics (beta) after it becomes generally available (GA). Redpanda will be unable to continue using data that was generated by the beta version of this feature. Also, any topic specified to be a Cloud Topic cannot subsequently be converted back to a standard, or non-Cloud Topic. == Create Cloud Topics @@ -48,7 +48,7 @@ To create a Cloud Topic: ---- rpk topic create -c redpanda.cloud_topic.enable=true ---- -+ + [,bash,role=no-copy] ---- TOPIC STATUS @@ -59,7 +59,7 @@ To enable a Cloud Topic in a cluster: [,bash] ---- -rpk topic create -c redpanda.cloud_topic.enabled=true +rpk cluster config set unstable_beta_feature_cloud_topics_enabled = true ---- // end::single-source[] \ No newline at end of file From 82b4be230646c2470eafa5c0c66d9589a27206fb Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Thu, 20 Nov 2025 13:26:28 -0500 Subject: [PATCH 21/75] WIP --- modules/develop/pages/manage-topics/index.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/develop/pages/manage-topics/index.adoc b/modules/develop/pages/manage-topics/index.adoc index 37897f4414..d1eb45c4f9 100644 --- a/modules/develop/pages/manage-topics/index.adoc +++ b/modules/develop/pages/manage-topics/index.adoc @@ -1,4 +1,4 @@ = Topics :page-categories: Clients, Development :description: Learn how to manage topics in Redpanda, including creation, configuration, and advanced features. -:page-layout: index \ No newline at end of file +:page-layout: index From d6e4097be27b3b264d206a0471916df09c396dc9 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Thu, 20 Nov 2025 13:26:48 -0500 Subject: [PATCH 22/75] WIP --- .../pages/manage-topics/cloud-topics.adoc | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index a6544ab912..8b7dce03d0 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -11,15 +11,15 @@ include::shared:partial$enterprise-license.adoc[] ==== endif::[] -While standard Kafka glossterm:topics[] are ideal for high-throughput, cost-sensitive workloads, Cloud Topics are optimized for high-throughput, cost-sensitive workloads. Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud object storage (S3, GCS, MinIO) as the primary mechanism for backing up data. The difference in response time is typically 500ms, which is only impactful for latency-sensitive applications. So when using Cloud Topics, you get the same durability that you have with standard topics, but you also eliminate the significant cross-AZ costs typically incurred when you have high-throughput workloads (for example, for audit logs). +Starting in v25.3, Redpanda provides Cloud Topics to support multi-modal streaming workloads in the most cost-effective way possible. While standard Kafka xref:config-topics.adoc[topics] are ideal for latency-sensitive, high-throughput workloads (for example, for audit logs), Cloud Topics are optimized for high-throughput, cost-sensitive workloads that can tolerate higher latencies. Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud object storage (S3, GCS, MinIO) as the primary mechanism for backing up data. The difference in response time is typically 500ms, which is only impactful for latency-sensitive applications. == Prerequisites -- rpk v25.3.1: See xref:get-started:rpk-install.adoc[]. +- xref:get-started:rpk-install.adoc[] v25.3 or later. ifndef::env-cloud[] - xref:manage:tiered-storage.adoc#set-up-tiered-storage[Set up] and xref:manage:tiered-storage.adoc#enable-tiered-storage[enable Tiered Storage] on your Redpanda cluster. -- xref:manage:tiered-storage.adoc#configure-object-storage[Configure object storage] -- include::shared:partial$enterprise-license.adoc[] +- xref:manage:tiered-storage.adoc#configure-object-storage[Configure object storage]. +- You must have an Enterprise license. To check if you already have a license key applied to your cluster: @@ -38,10 +38,16 @@ Not all Redpanda features are available when using Cloud Topics (beta). The foll - Topic recovery - Remote Read Replicas +Also note that Cloud Topics do not work in simple Docker containers unless Cloud storage is set up in them. + IMPORTANT: You should expect to throw away any cluster running the Cloud Topics (beta) after it becomes generally available (GA). Redpanda will be unable to continue using data that was generated by the beta version of this feature. Also, any topic specified to be a Cloud Topic cannot subsequently be converted back to a standard, or non-Cloud Topic. == Create Cloud Topics +You can make a topic a Cloud Topic only at creation time. This is a two-step process where you first create the Cloud Topic, then enable that Cloud Topic in your cluster. + +IMPORTANT: Once you create a Cloud Topic, you cannot subsequently convert it back into a standard Kafka topic. + To create a Cloud Topic: [,bash] @@ -59,7 +65,9 @@ To enable a Cloud Topic in a cluster: [,bash] ---- -rpk cluster config set unstable_beta_feature_cloud_topics_enabled = true +rpk cluster config set unstable_beta_feature_cloud_topics_enabled=true ---- +NOTE: For non-replicated cross-AZ traffic (for example, for producers), if your throughput costs remain elevated, you can use xref:../produce-data/leader-pinning.adoc[leader pinning], which positions a topic's partition leader close to the producers, and helps eliminate this traffic + // end::single-source[] \ No newline at end of file From d2143196902b3d35a8df92f1d3cb5044524a789e Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Thu, 20 Nov 2025 14:33:51 -0500 Subject: [PATCH 23/75] wip --- modules/develop/pages/manage-topics/cloud-topics.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index 8b7dce03d0..a317df9422 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -68,6 +68,6 @@ To enable a Cloud Topic in a cluster: rpk cluster config set unstable_beta_feature_cloud_topics_enabled=true ---- -NOTE: For non-replicated cross-AZ traffic (for example, for producers), if your throughput costs remain elevated, you can use xref:../produce-data/leader-pinning.adoc[leader pinning], which positions a topic's partition leader close to the producers, and helps eliminate this traffic +NOTE: For non-replicated cross-AZ traffic (for example, for producers), if your throughput costs remain elevated, you can use xref:../produce-data/leader-pinning.adoc[leader pinning], which positions a topic's partition leader close to the producers, and helps eliminate this traffic. // end::single-source[] \ No newline at end of file From cb2c6a9c393cc4f3c93352e8c563d0d569f55fa9 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Thu, 20 Nov 2025 15:20:58 -0500 Subject: [PATCH 24/75] fix change in config topics --- modules/develop/pages/manage-topics/config-topics.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/develop/pages/manage-topics/config-topics.adoc b/modules/develop/pages/manage-topics/config-topics.adoc index 9c361c641b..91515616c9 100644 --- a/modules/develop/pages/manage-topics/config-topics.adoc +++ b/modules/develop/pages/manage-topics/config-topics.adoc @@ -225,7 +225,7 @@ CONFIGS KEY VALUE SOURCE cleanup.policy delete DYNAMIC_TOPIC_CONFIG compression.type producer DEFAULT_CONFIG -max.message.bytes 1048576 DEFAULT_CONFIG +max.message.bytes 20971520 DEFAULT_CONFIG message.timestamp.type CreateTime DEFAULT_CONFIG redpanda.datapolicy function_name: script_name: DEFAULT_CONFIG redpanda.remote.delete true DEFAULT_CONFIG From 53a6f538c3fe3858db20510823c145c43491ae3d Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Thu, 20 Nov 2025 18:55:00 -0500 Subject: [PATCH 25/75] WIP --- .../pages/manage-topics/cloud-topics.adoc | 44 +++++++++++++------ 1 file changed, 30 insertions(+), 14 deletions(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index a317df9422..ba0e5df425 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -1,7 +1,6 @@ = Cloud Topics :description: Cloud Topics are Kafka topics that enable users to trade off latency for lower costs. :page-beta: true -:page-categories: Clients, Development // tag::single-source[] ifndef::env-cloud[] @@ -18,6 +17,17 @@ Starting in v25.3, Redpanda provides Cloud Topics to support multi-modal streami - xref:get-started:rpk-install.adoc[] v25.3 or later. ifndef::env-cloud[] - xref:manage:tiered-storage.adoc#set-up-tiered-storage[Set up] and xref:manage:tiered-storage.adoc#enable-tiered-storage[enable Tiered Storage] on your Redpanda cluster. ++ +[NOTE] +==== +Be sure to set the following cluster-level properties to `false`: + +* config_ref:cloud_storage_enable_remote_write,false,properties/object-storage-properties[] +* config_ref:cloud_storage_enable_remote_read,false,properties/object-storage-properties[] + +For details, see xref:manage:tiered-storage.adoc#enable-tiered-storage-for-a-cluster[Enable Tiered Storage for a cluster]. +==== + - xref:manage:tiered-storage.adoc#configure-object-storage[Configure object storage]. - You must have an Enterprise license. @@ -40,32 +50,38 @@ Not all Redpanda features are available when using Cloud Topics (beta). The foll Also note that Cloud Topics do not work in simple Docker containers unless Cloud storage is set up in them. -IMPORTANT: You should expect to throw away any cluster running the Cloud Topics (beta) after it becomes generally available (GA). Redpanda will be unable to continue using data that was generated by the beta version of this feature. Also, any topic specified to be a Cloud Topic cannot subsequently be converted back to a standard, or non-Cloud Topic. +IMPORTANT: You should expect to throw away any cluster running Cloud Topics (beta) after it becomes generally available (GA). Redpanda will be unable to continue using data that was generated by the beta version of this feature. Also, any topic specified to be a Cloud Topic cannot subsequently be converted back to a standard Kafka topic. + +== Set up Cloud Topics + +To set up Cloud Topics: -== Create Cloud Topics +. Enable the Cloud Topic for your cluster. -You can make a topic a Cloud Topic only at creation time. This is a two-step process where you first create the Cloud Topic, then enable that Cloud Topic in your cluster. +. Create the Cloud Topic. + +You can make a topic a Cloud Topic only at creation time. IMPORTANT: Once you create a Cloud Topic, you cannot subsequently convert it back into a standard Kafka topic. -To create a Cloud Topic: +=== Enable Cloud Topics for your cluster -[,bash] ----- -rpk topic create -c redpanda.cloud_topic.enable=true ----- +Run the `rpk cluster config set` command, then edit the following required property: -[,bash,role=no-copy] +[,properties] ---- -TOPIC STATUS -audit.analytics.may2025 OK +unstable_beta_feature_cloud_topics_enabled: true ---- -To enable a Cloud Topic in a cluster: +=== Create a Cloud Topic + +Run the following command to create a Cloud Topic: [,bash] ---- -rpk cluster config set unstable_beta_feature_cloud_topics_enabled=true +rpk topic create -c redpanda.cloud_topic.enable=true +TOPIC STATUS +audit.analytics.may2025 OK ---- NOTE: For non-replicated cross-AZ traffic (for example, for producers), if your throughput costs remain elevated, you can use xref:../produce-data/leader-pinning.adoc[leader pinning], which positions a topic's partition leader close to the producers, and helps eliminate this traffic. From 842e4b1b7e183937096ed7a1100f923a634dc3a7 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Thu, 20 Nov 2025 19:05:39 -0500 Subject: [PATCH 26/75] wip --- modules/develop/pages/manage-topics/cloud-topics.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index ba0e5df425..146df6097e 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -20,7 +20,7 @@ ifndef::env-cloud[] + [NOTE] ==== -Be sure to set the following cluster-level properties to `false`: +Be sure to set the following cluster-level properties to `false` to disable Tiered Storage for new topics: * config_ref:cloud_storage_enable_remote_write,false,properties/object-storage-properties[] * config_ref:cloud_storage_enable_remote_read,false,properties/object-storage-properties[] From 3ebce8d06c391c0d30e01d2c3bd1ad71175406c7 Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Fri, 21 Nov 2025 13:17:23 -0600 Subject: [PATCH 27/75] Update modules/develop/pages/manage-topics/cloud-topics.adoc --- modules/develop/pages/manage-topics/cloud-topics.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index 146df6097e..606145edca 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -1,5 +1,5 @@ = Cloud Topics -:description: Cloud Topics are Kafka topics that enable users to trade off latency for lower costs. +:description: Cloud Topics are Redpanda topics that enable users to trade off latency for lower costs. :page-beta: true // tag::single-source[] From 51327726af25c7dbfd3aceae01205706862d3a14 Mon Sep 17 00:00:00 2001 From: JakeSCahill Date: Tue, 25 Nov 2025 10:43:20 +0000 Subject: [PATCH 28/75] Add cloud_topics configs --- docs-data/property-overrides.json | 15 ++++----------- .../attachments/redpanda-properties-v25.3.7.json | 15 ++++----------- .../partials/properties/cluster-properties.adoc | 15 +-------------- .../partials/properties/topic-properties.adoc | 6 +++--- 4 files changed, 12 insertions(+), 39 deletions(-) diff --git a/docs-data/property-overrides.json b/docs-data/property-overrides.json index 43744c347b..a220a7b5a6 100644 --- a/docs-data/property-overrides.json +++ b/docs-data/property-overrides.json @@ -469,11 +469,9 @@ "config_scope": "cluster" }, "cloud_topics_disable_reconciliation_loop": { - "exclude_from_docs": true, "config_scope": "cluster" }, "cloud_topics_enabled": { - "exclude_from_docs": true, "config_scope": "cluster" }, "cloud_topics_epoch_service_epoch_increment_interval": { @@ -485,23 +483,18 @@ "version": "v25.3.3" }, "cloud_topics_long_term_garbage_collection_interval": { - "exclude_from_docs": true, "config_scope": "cluster" }, "cloud_topics_produce_batching_size_threshold": { - "exclude_from_docs": true, "config_scope": "cluster" }, "cloud_topics_produce_cardinality_threshold": { - "exclude_from_docs": true, "config_scope": "cluster" }, "cloud_topics_produce_upload_interval": { - "exclude_from_docs": true, "config_scope": "cluster" }, "cloud_topics_reconciliation_interval": { - "exclude_from_docs": true, "config_scope": "cluster" }, "cloud_topics_short_term_gc_backoff_interval": { @@ -1709,8 +1702,8 @@ "xref:reference:rpk/rpk-topic/rpk-topic-describe.adoc[`rpk topic describe`]", "xref:reference:rpk/rpk-topic/rpk-topic-alter-config.adoc[`rpk topic alter-config`]", "xref:reference:properties/cluster-properties.adoc#default_topic_replication[`default_topic_replication`]", - "xref:develop:config-topics.adoc#choose-the-replication-factor[Choose the replication factor]", - "xref:develop:config-topics.adoc#change-the-replication-factor[Change the replication factor]", + "xref:develop:manage-topics.adoc#choose-the-replication-factor[Choose the replication factor]", + "xref:develop:manage-topics.adoc#change-the-replication-factor[Change the replication factor]", "xref:reference:properties/cluster-properties.adoc#default_topic_replication[default_topic_replication]" ], "config_scope": "topic" @@ -2091,7 +2084,7 @@ "write.caching": { "description": "The write caching mode to apply to a topic.\n\nWhen `write.caching` is set, it overrides the cluster property xref:cluster-properties.adoc#write_caching_default[`write_caching_default`]. Write caching acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to be written to disk. With `acks=all`, this provides lower latency while still ensuring that a majority of brokers acknowledge the write. Fsyncs follow <> and <>, whichever is reached first.", "related_topics": [ - "xref:develop:config-topics.adoc#configure-write-caching[Write caching]", + "xref:develop:manage-topics.adoc#configure-write-caching[Write caching]", "xref:manage:tiered-storage.adoc[Tiered Storage]", "xref:reference:properties/cluster-properties.adoc#write_caching_default[`write_caching_default`]", "xref:cluster-properties.adoc#write_caching_default[`write_caching_default`]" @@ -2101,7 +2094,7 @@ "write_caching_default": { "related_topics": [ "xref:reference:properties/topic-properties.adoc#writecaching[`write.caching`]", - "xref:develop:config-topics.adoc#configure-write-caching[Write caching]" + "xref:develop:manage-topics.adoc#configure-write-caching[Write caching]" ], "config_scope": "cluster", "description": "The default write caching mode to apply to user topics. Write caching acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to be written to disk. With `acks=all`, this provides lower latency while still ensuring that a majority of brokers acknowledge the write. \n\nFsyncs follow <> and <>, whichever is reached first.\n\nThe `write_caching_default` cluster property can be overridden with the xref:reference:properties/topic-properties.adoc#writecaching[`write.caching`] topic property." diff --git a/modules/reference/attachments/redpanda-properties-v25.3.7.json b/modules/reference/attachments/redpanda-properties-v25.3.7.json index 0e9b183d44..8a32109929 100644 --- a/modules/reference/attachments/redpanda-properties-v25.3.7.json +++ b/modules/reference/attachments/redpanda-properties-v25.3.7.json @@ -3207,7 +3207,6 @@ "default": false, "defined_in": "src/v/config/configuration.cc", "description": "Disables the cloud topics reconciliation loop. Disabling the loop can negatively impact performance and stability of the cluster.", - "exclude_from_docs": true, "is_deprecated": false, "is_enterprise": false, "name": "cloud_topics_disable_reconciliation_loop", @@ -3233,7 +3232,6 @@ "enterprise_value": [ "true" ], - "exclude_from_docs": true, "is_deprecated": false, "is_enterprise": true, "name": "cloud_topics_enabled", @@ -3297,7 +3295,6 @@ "default_human_readable": "5 minutes", "defined_in": "src/v/config/configuration.cc", "description": "Time interval after which data is garbage collected from long term storage.", - "exclude_from_docs": true, "is_deprecated": false, "is_enterprise": false, "maximum": 17592186044415, @@ -3318,7 +3315,6 @@ "default": 4194304, "defined_in": "src/v/config/configuration.cc", "description": "The size limit for the object size in cloud topics. When the amount of data on a shard reaches this limit, an upload is triggered.", - "exclude_from_docs": true, "is_deprecated": false, "is_enterprise": false, "name": "cloud_topics_produce_batching_size_threshold", @@ -3337,7 +3333,6 @@ "default": 1000, "defined_in": "src/v/config/configuration.cc", "description": "Threshold for the object cardinality in cloud topics. When the number of partitions in waiting for the upload reach this limit, an upload is triggered.", - "exclude_from_docs": true, "is_deprecated": false, "is_enterprise": false, "name": "cloud_topics_produce_cardinality_threshold", @@ -3357,7 +3352,6 @@ "default_human_readable": "250 milliseconds", "defined_in": "src/v/config/configuration.cc", "description": "Time interval after which the upload is triggered.", - "exclude_from_docs": true, "is_deprecated": false, "is_enterprise": false, "maximum": 17592186044415, @@ -3379,7 +3373,6 @@ "default_human_readable": "10 seconds", "defined_in": "src/v/config/configuration.cc", "description": "Time interval after which data is moved from short term storage to long term storage.", - "exclude_from_docs": true, "is_deprecated": false, "is_enterprise": false, "maximum": 17592186044415, @@ -11512,8 +11505,8 @@ "xref:reference:rpk/rpk-topic/rpk-topic-describe.adoc[`rpk topic describe`]", "xref:reference:rpk/rpk-topic/rpk-topic-alter-config.adoc[`rpk topic alter-config`]", "xref:reference:properties/cluster-properties.adoc#default_topic_replication[`default_topic_replication`]", - "xref:develop:config-topics.adoc#choose-the-replication-factor[Choose the replication factor]", - "xref:develop:config-topics.adoc#change-the-replication-factor[Change the replication factor]", + "xref:develop:manage-topics.adoc#choose-the-replication-factor[Choose the replication factor]", + "xref:develop:manage-topics.adoc#change-the-replication-factor[Change the replication factor]", "xref:reference:properties/cluster-properties.adoc#default_topic_replication[default_topic_replication]" ], "type": "integer" @@ -13886,7 +13879,7 @@ "name": "write.caching", "needs_restart": false, "related_topics": [ - "xref:develop:config-topics.adoc#configure-write-caching[Write caching]", + "xref:develop:manage-topics.adoc#configure-write-caching[Write caching]", "xref:manage:tiered-storage.adoc[Tiered Storage]", "xref:reference:properties/cluster-properties.adoc#write_caching_default[`write_caching_default`]", "xref:cluster-properties.adoc#write_caching_default[`write_caching_default`]" @@ -13916,7 +13909,7 @@ "nullable": false, "related_topics": [ "xref:reference:properties/topic-properties.adoc#writecaching[`write.caching`]", - "xref:develop:config-topics.adoc#configure-write-caching[Write caching]" + "xref:develop:manage-topics.adoc#configure-write-caching[Write caching]" ], "type": "string", "visibility": "user" diff --git a/modules/reference/partials/properties/cluster-properties.adoc b/modules/reference/partials/properties/cluster-properties.adoc index 5443c0734f..0e2c29f5c1 100644 --- a/modules/reference/partials/properties/cluster-properties.adoc +++ b/modules/reference/partials/properties/cluster-properties.adoc @@ -860,7 +860,6 @@ endif::[] // end::redpanda-cloud[] -// tag::exclude-from-docs[] === cloud_topics_disable_reconciliation_loop Disables the cloud topics reconciliation loop. Disabling the loop can negatively impact performance and stability of the cluster. @@ -901,9 +900,7 @@ endif::[] |=== -// end::exclude-from-docs[] -// tag::exclude-from-docs[] === cloud_topics_enabled Enable cloud topics. @@ -954,7 +951,6 @@ endif::[] |=== -// end::exclude-from-docs[] === cloud_topics_epoch_service_epoch_increment_interval @@ -1104,9 +1100,7 @@ endif::[] |=== -// end::exclude-from-docs[] -// tag::exclude-from-docs[] === cloud_topics_produce_batching_size_threshold The size limit for the object size in cloud topics. When the amount of data on a shard reaches this limit, an upload is triggered. @@ -1147,9 +1141,7 @@ endif::[] |=== -// end::exclude-from-docs[] -// tag::exclude-from-docs[] === cloud_topics_produce_cardinality_threshold Threshold for the object cardinality in cloud topics. When the number of partitions in waiting for the upload reach this limit, an upload is triggered. @@ -1190,9 +1182,7 @@ endif::[] |=== -// end::exclude-from-docs[] -// tag::exclude-from-docs[] === cloud_topics_produce_upload_interval Time interval after which the upload is triggered. @@ -1236,9 +1226,7 @@ endif::[] |=== -// end::exclude-from-docs[] -// tag::exclude-from-docs[] === cloud_topics_reconciliation_interval Time interval after which data is moved from short term storage to long term storage. @@ -1282,7 +1270,6 @@ endif::[] |=== -// end::exclude-from-docs[] === cloud_topics_short_term_gc_backoff_interval @@ -19931,7 +19918,7 @@ endif::[] | * xref:reference:properties/topic-properties.adoc#writecaching[`write.caching`] -* xref:develop:config-topics.adoc#configure-write-caching[Write caching] +* xref:develop:manage-topics.adoc#configure-write-caching[Write caching] |=== diff --git a/modules/reference/partials/properties/topic-properties.adoc b/modules/reference/partials/properties/topic-properties.adoc index 1463d30bfd..1278f061e3 100644 --- a/modules/reference/partials/properties/topic-properties.adoc +++ b/modules/reference/partials/properties/topic-properties.adoc @@ -1710,9 +1710,9 @@ endif::[] * xref:reference:properties/cluster-properties.adoc#default_topic_replication[`default_topic_replication`] -* xref:develop:config-topics.adoc#choose-the-replication-factor[Choose the replication factor] +* xref:develop:manage-topics.adoc#choose-the-replication-factor[Choose the replication factor] -* xref:develop:config-topics.adoc#change-the-replication-factor[Change the replication factor] +* xref:develop:manage-topics.adoc#change-the-replication-factor[Change the replication factor] * xref:reference:properties/cluster-properties.adoc#default_topic_replication[default_topic_replication] @@ -2106,7 +2106,7 @@ endif::[] | Related topics | -* xref:develop:config-topics.adoc#configure-write-caching[Write caching] +* xref:develop:manage-topics.adoc#configure-write-caching[Write caching] * xref:manage:tiered-storage.adoc[Tiered Storage] From affffe0653b4deaf217b39e336e2f0e8f25b73ec Mon Sep 17 00:00:00 2001 From: JakeSCahill Date: Tue, 25 Nov 2025 16:00:56 +0000 Subject: [PATCH 29/75] Add overrides --- docs-data/property-overrides.json | 24 +++++++++++++-- .../redpanda-properties-v25.3.7.json | 26 ++++++++++++---- .../properties/cluster-properties.adoc | 30 +++++++++++++++++-- .../partials/properties/topic-properties.adoc | 11 +++++-- 4 files changed, 78 insertions(+), 13 deletions(-) diff --git a/docs-data/property-overrides.json b/docs-data/property-overrides.json index a220a7b5a6..e6fa76eaab 100644 --- a/docs-data/property-overrides.json +++ b/docs-data/property-overrides.json @@ -472,6 +472,11 @@ "config_scope": "cluster" }, "cloud_topics_enabled": { + "description": "Enable Cloud Topics for the cluster. Cloud Topics are optimized for high-throughput, cost-sensitive workloads that can tolerate higher latencies compared to standard Kafka topics.", + "related_topics": [ + "xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics]", + "xref:reference:properties/topic-properties.adoc#redpandacloudtopicenabled[`redpanda.cloud_topic.enabled`]" + ], "config_scope": "cluster" }, "cloud_topics_epoch_service_epoch_increment_interval": { @@ -492,9 +497,20 @@ "config_scope": "cluster" }, "cloud_topics_produce_upload_interval": { + "description": "Time interval after which data is uploaded to object storage for Cloud Topics. When this time threshold is reached, Redpanda triggers an upload of buffered data to the cloud object storage backend (S3, GCS, or MinIO), regardless of whether the size or cardinality thresholds have been met.\n\nThis property works together with <> and <> to control when uploads occur. An upload is triggered when any of these three thresholds is reached.", + "related_topics": [ + "xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics]", + "xref:reference:properties/cluster-properties.adoc#cloud_topics_produce_batching_size_threshold[`cloud_topics_produce_batching_size_threshold`]", + "xref:reference:properties/cluster-properties.adoc#cloud_topics_produce_cardinality_threshold[`cloud_topics_produce_cardinality_threshold`]" + ], "config_scope": "cluster" }, "cloud_topics_reconciliation_interval": { + "description": "Time interval at which Redpanda reconciles data between short-term local storage and long-term cloud object storage for Cloud Topics. During reconciliation, Redpanda moves data that has been successfully persisted to object storage from local storage, freeing up local disk space while maintaining data durability in the cloud.\n\nThis reconciliation process is essential for Cloud Topics to achieve their cost-efficiency goals by minimizing the amount of expensive local storage required, while ensuring data remains accessible from durable cloud storage.", + "related_topics": [ + "xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics]", + "xref:reference:properties/cluster-properties.adoc#cloud_topics_long_term_garbage_collection_interval[`cloud_topics_long_term_garbage_collection_interval`]" + ], "config_scope": "cluster" }, "cloud_topics_short_term_gc_backoff_interval": { @@ -1566,9 +1582,13 @@ "category": "redpanda" }, "redpanda.cloud_topic.enabled": { + "description": "Enable this topic as a Cloud Topic. Cloud Topics are optimized for high-throughput, cost-sensitive workloads that can tolerate higher latencies compared to standard Kafka topics.\n\nIMPORTANT: You can only set this property when creating a topic. Once a topic is created as a Cloud Topic, it cannot be converted back to a standard Kafka topic. Before creating Cloud Topics, you must enable the cluster-wide <> property.", + "related_topics": [ + "xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics]", + "xref:reference:properties/cluster-properties.adoc#cloud_topics_enabled[`cloud_topics_enabled`]" + ], "config_scope": "topic", - "category": "tiered-storage", - "exclude_from_docs": true + "category": "tiered-storage" }, "redpanda.iceberg.delete": { "description": "Whether the corresponding Iceberg table is deleted upon deleting the topic.", diff --git a/modules/reference/attachments/redpanda-properties-v25.3.7.json b/modules/reference/attachments/redpanda-properties-v25.3.7.json index 8a32109929..adce09d41c 100644 --- a/modules/reference/attachments/redpanda-properties-v25.3.7.json +++ b/modules/reference/attachments/redpanda-properties-v25.3.7.json @@ -3224,7 +3224,7 @@ "config_scope": "cluster", "default": false, "defined_in": "src/v/config/configuration.cc", - "description": "Enable cloud topics.", + "description": "Enable Cloud Topics for the cluster. Cloud Topics are optimized for high-throughput, cost-sensitive workloads that can tolerate higher latencies compared to standard Kafka topics.", "enterprise_constructor": "restricted_only", "enterprise_restricted_value": [ "true" @@ -3237,6 +3237,10 @@ "name": "cloud_topics_enabled", "needs_restart": false, "nullable": false, + "related_topics": [ + "xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics]", + "xref:reference:properties/topic-properties.adoc#redpandacloudtopicenabled[`redpanda.cloud_topic.enabled`]" + ], "type": "boolean", "visibility": "user" }, @@ -3351,7 +3355,7 @@ "default": 250, "default_human_readable": "250 milliseconds", "defined_in": "src/v/config/configuration.cc", - "description": "Time interval after which the upload is triggered.", + "description": "Time interval after which data is uploaded to object storage for Cloud Topics. When this time threshold is reached, Redpanda triggers an upload of buffered data to the cloud object storage backend (S3, GCS, or MinIO), regardless of whether the size or cardinality thresholds have been met.\n\nThis property works together with <> and <> to control when uploads occur. An upload is triggered when any of these three thresholds is reached.", "is_deprecated": false, "is_enterprise": false, "maximum": 17592186044415, @@ -3359,6 +3363,11 @@ "name": "cloud_topics_produce_upload_interval", "needs_restart": false, "nullable": false, + "related_topics": [ + "xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics]", + "xref:reference:properties/cluster-properties.adoc#cloud_topics_produce_batching_size_threshold[`cloud_topics_produce_batching_size_threshold`]", + "xref:reference:properties/cluster-properties.adoc#cloud_topics_produce_cardinality_threshold[`cloud_topics_produce_cardinality_threshold`]" + ], "type": "integer", "visibility": "user" }, @@ -3372,7 +3381,7 @@ "default": 10000, "default_human_readable": "10 seconds", "defined_in": "src/v/config/configuration.cc", - "description": "Time interval after which data is moved from short term storage to long term storage.", + "description": "Time interval at which Redpanda reconciles data between short-term local storage and long-term cloud object storage for Cloud Topics. During reconciliation, Redpanda moves data that has been successfully persisted to object storage from local storage, freeing up local disk space while maintaining data durability in the cloud.\n\nThis reconciliation process is essential for Cloud Topics to achieve their cost-efficiency goals by minimizing the amount of expensive local storage required, while ensuring data remains accessible from durable cloud storage.", "is_deprecated": false, "is_enterprise": false, "maximum": 17592186044415, @@ -3380,6 +3389,10 @@ "name": "cloud_topics_reconciliation_interval", "needs_restart": false, "nullable": false, + "related_topics": [ + "xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics]", + "xref:reference:properties/cluster-properties.adoc#cloud_topics_long_term_garbage_collection_interval[`cloud_topics_long_term_garbage_collection_interval`]" + ], "type": "integer", "visibility": "tunable" }, @@ -11014,13 +11027,16 @@ "corresponding_cluster_property": null, "default": null, "defined_in": "src/v/kafka/protocol/topic_properties.h", - "description": "", - "exclude_from_docs": true, + "description": "Enable this topic as a Cloud Topic. Cloud Topics are optimized for high-throughput, cost-sensitive workloads that can tolerate higher latencies compared to standard Kafka topics.\n\nIMPORTANT: You can only set this property when creating a topic. Once a topic is created as a Cloud Topic, it cannot be converted back to a standard Kafka topic. Before creating Cloud Topics, you must enable the cluster-wide <> property.", "is_deprecated": false, "is_enterprise": false, "is_topic_property": true, "name": "redpanda.cloud_topic.enabled", "needs_restart": false, + "related_topics": [ + "xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics]", + "xref:reference:properties/cluster-properties.adoc#cloud_topics_enabled[`cloud_topics_enabled`]" + ], "type": "string" }, "redpanda.iceberg.delete": { diff --git a/modules/reference/partials/properties/cluster-properties.adoc b/modules/reference/partials/properties/cluster-properties.adoc index 0e2c29f5c1..8b2df4e98a 100644 --- a/modules/reference/partials/properties/cluster-properties.adoc +++ b/modules/reference/partials/properties/cluster-properties.adoc @@ -903,7 +903,7 @@ endif::[] === cloud_topics_enabled -Enable cloud topics. +Enable Cloud Topics for the cluster. Cloud Topics are optimized for high-throughput, cost-sensitive workloads that can tolerate higher latencies compared to standard Kafka topics. ifndef::env-cloud[] .Enterprise license required @@ -949,6 +949,12 @@ ifndef::env-cloud[] | User endif::[] +| Related topics +| +* xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics] + +* xref:reference:properties/topic-properties.adoc#redpandacloudtopicenabled[`redpanda.cloud_topic.enabled`] + |=== @@ -1185,7 +1191,9 @@ endif::[] === cloud_topics_produce_upload_interval -Time interval after which the upload is triggered. +Time interval after which data is uploaded to object storage for Cloud Topics. When this time threshold is reached, Redpanda triggers an upload of buffered data to the cloud object storage backend (S3, GCS, or MinIO), regardless of whether the size or cardinality thresholds have been met. + +This property works together with <> and <> to control when uploads occur. An upload is triggered when any of these three thresholds is reached. [cols="1s,2a"] |=== @@ -1224,12 +1232,22 @@ ifndef::env-cloud[] | User endif::[] +| Related topics +| +* xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics] + +* xref:reference:properties/cluster-properties.adoc#cloud_topics_produce_batching_size_threshold[`cloud_topics_produce_batching_size_threshold`] + +* xref:reference:properties/cluster-properties.adoc#cloud_topics_produce_cardinality_threshold[`cloud_topics_produce_cardinality_threshold`] + |=== === cloud_topics_reconciliation_interval -Time interval after which data is moved from short term storage to long term storage. +Time interval at which Redpanda reconciles data between short-term local storage and long-term cloud object storage for Cloud Topics. During reconciliation, Redpanda moves data that has been successfully persisted to object storage from local storage, freeing up local disk space while maintaining data durability in the cloud. + +This reconciliation process is essential for Cloud Topics to achieve their cost-efficiency goals by minimizing the amount of expensive local storage required, while ensuring data remains accessible from durable cloud storage. [cols="1s,2a"] |=== @@ -1268,6 +1286,12 @@ ifndef::env-cloud[] | Tunable endif::[] +| Related topics +| +* xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics] + +* xref:reference:properties/cluster-properties.adoc#cloud_topics_long_term_garbage_collection_interval[`cloud_topics_long_term_garbage_collection_interval`] + |=== diff --git a/modules/reference/partials/properties/topic-properties.adoc b/modules/reference/partials/properties/topic-properties.adoc index 1278f061e3..69126c6789 100644 --- a/modules/reference/partials/properties/topic-properties.adoc +++ b/modules/reference/partials/properties/topic-properties.adoc @@ -945,11 +945,11 @@ endif::[] // end::category-retention-compaction[] // tag::category-tiered-storage[] -// tag::exclude-from-docs[] === redpanda.cloud_topic.enabled -No description available. +Enable this topic as a Cloud Topic. Cloud Topics are optimized for high-throughput, cost-sensitive workloads that can tolerate higher latencies compared to standard Kafka topics. +IMPORTANT: You can only set this property when creating a topic. Once a topic is created as a Cloud Topic, it cannot be converted back to a standard Kafka topic. Before creating Cloud Topics, you must enable the cluster-wide <> property. [cols="1s,2a"] |=== @@ -977,9 +977,14 @@ ifndef::env-cloud[] | Yes endif::[] +| Related topics +| +* xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics] + +* xref:reference:properties/cluster-properties.adoc#cloud_topics_enabled[`cloud_topics_enabled`] + |=== -// end::exclude-from-docs[] // end::category-tiered-storage[] // tag::category-iceberg-integration[] From c27c1ae92046a6132f61d41d986017d052d2a80c Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 25 Nov 2025 15:25:51 -0500 Subject: [PATCH 30/75] implemented review feedback --- .../pages/manage-topics/cloud-topics.adoc | 63 +++++++++++-------- 1 file changed, 36 insertions(+), 27 deletions(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index 606145edca..8d3e3cecf5 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -10,7 +10,15 @@ include::shared:partial$enterprise-license.adoc[] ==== endif::[] -Starting in v25.3, Redpanda provides Cloud Topics to support multi-modal streaming workloads in the most cost-effective way possible. While standard Kafka xref:config-topics.adoc[topics] are ideal for latency-sensitive, high-throughput workloads (for example, for audit logs), Cloud Topics are optimized for high-throughput, cost-sensitive workloads that can tolerate higher latencies. Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud object storage (S3, GCS, MinIO) as the primary mechanism for backing up data. The difference in response time is typically 500ms, which is only impactful for latency-sensitive applications. +Starting in v25.3, Redpanda provides Cloud Topics to support multi-modal streaming workloads in the most cost-effective way possible: as a configurable option on a single cluster. While standard Redpanda xref:config-topics.adoc[topics] that use local storage or Tiered Storage are ideal for latency-sensitive, low-throughput workloads (for example, for audit logs), Cloud Topics are optimized for latency-tolerant, high-throughput workloads where cloud provider infrastructure costs (especially cross-AZ networking charges) are a major consideration that can become the dominant cost driver at high throughput. These workloads can include observability streams, offline analytics, AI/ML model training data feeds, or development environments where streaming latency isn't critical. + +Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud object storage (S3, ADLS,GCS, MinIO) as the primary mechanism for backing up data. This eliminates over 90% of the cost of replicating data over network links in multi-AZ or multi-region clusters. The difference in end-to-end latency is typically between 500ms-1s, which is often acceptable for many streaming workloads, and can unlock new streaming use cases that were previously not cost effective. + +:tip-caption: Cost savings example + +TIP: In a cloud environment tested by Redpanda, the networking costs of data replication to run a single 100MBps workload were reduced from over $10,000 per month to $80 per month. While this does not represent 100% of total operating costs (Kafka batch sizes can affect this cost reduction slightly), at high throughputs this savings still accounts for the majority of cloud provider expense, reducing the total infrastructure cost of running Kafka workloads by an order of magnitude or more. + +:tip-caption: Tip == Prerequisites @@ -20,23 +28,37 @@ ifndef::env-cloud[] + [NOTE] ==== -Be sure to set the following cluster-level properties to `false` to disable Tiered Storage for new topics: +When you enabled Tiered Storage at the cluster level, the default is that all new topics are Tiered Storage (or standard) topics. However, if you plan to use Cloud Topics for all new topics in a Redpanda cluster, be sure to set the following cluster-level properties to `false`: * config_ref:cloud_storage_enable_remote_write,false,properties/object-storage-properties[] * config_ref:cloud_storage_enable_remote_read,false,properties/object-storage-properties[] +This ensures that newly-created Redpanda topics are Cloud Topics by default. + For details, see xref:manage:tiered-storage.adoc#enable-tiered-storage-for-a-cluster[Enable Tiered Storage for a cluster]. ==== - xref:manage:tiered-storage.adoc#configure-object-storage[Configure object storage]. - You must have an Enterprise license. -To check if you already have a license key applied to your cluster: +To check your license status: [,bash] ---- rpk cluster license info ---- + +If you intend to use Docker Compose while working with Cloud Topics, see the https://docs.docker.com/compose/install/[Docker Compose installation documentation^]. To verify you have it installed, run: + +[source,bash] +---- +docker compose version +---- + +You should see the installed Docker Compose version. + +For more details about using Docker Compose with Redpanda, see the xref:get-started:quick-start.adoc[Redpanda Self-Managed Quickstart]. + endif::[] == Limitations @@ -48,42 +70,29 @@ Not all Redpanda features are available when using Cloud Topics (beta). The foll - Topic recovery - Remote Read Replicas -Also note that Cloud Topics do not work in simple Docker containers unless Cloud storage is set up in them. - -IMPORTANT: You should expect to throw away any cluster running Cloud Topics (beta) after it becomes generally available (GA). Redpanda will be unable to continue using data that was generated by the beta version of this feature. Also, any topic specified to be a Cloud Topic cannot subsequently be converted back to a standard Kafka topic. - -== Set up Cloud Topics - -To set up Cloud Topics: - -. Enable the Cloud Topic for your cluster. - -. Create the Cloud Topic. +IMPORTANT: You should expect to throw away any cluster running Cloud Topics (beta) after it becomes generally available (GA). Redpanda may be unable to continue using Cloud Topics created with the beta version, and you may be unable to upgrade to future versions. Also, any topic specified to be a Cloud Topic cannot be subsequently converted back to a standard Redpanda topic that uses local or Tiered Storage. -You can make a topic a Cloud Topic only at creation time. +== Enable Cloud Topics for your cluster -IMPORTANT: Once you create a Cloud Topic, you cannot subsequently convert it back into a standard Kafka topic. +You must first enable Cloud Topics for your cluster: -=== Enable Cloud Topics for your cluster - -Run the `rpk cluster config set` command, then edit the following required property: - -[,properties] +[,bash] ---- -unstable_beta_feature_cloud_topics_enabled: true +rpk cluster config set unstable_beta_feature_cloud_topics_enabled=true +Successfully updated configuration. New configuration version is 3. ---- -=== Create a Cloud Topic - -Run the following command to create a Cloud Topic: +Once enabled, create the Cloud Topic: [,bash] ---- -rpk topic create -c redpanda.cloud_topic.enable=true +rpk topic create -c cloud_topics_enabled=true TOPIC STATUS audit.analytics.may2025 OK ---- -NOTE: For non-replicated cross-AZ traffic (for example, for producers), if your throughput costs remain elevated, you can use xref:../produce-data/leader-pinning.adoc[leader pinning], which positions a topic's partition leader close to the producers, and helps eliminate this traffic. +You can make a topic a Cloud Topic only at topic creation time. + +NOTE: In addition to replication, cross-AZ ingress (Producer) and egress (Consumer) traffic can also contribute substantially to cloud networking cots. When running multi-AZ clusters in general, Redpanda strongly recommends using xref:../consume-data/follower-fetching.adoc[Follower Fetching], which allows consumers to avoid crossing network zones. When possible, you can use xref:../produce-data/leader-pinning.adoc[leader pinning], which positions a topic's partition leader close to the producers, providing a similar benefit for ingress traffic. These features can add additional savings to the replication cost savings of Cloud Topics. // end::single-source[] \ No newline at end of file From 28d23fbc480fd809406dc8c43caf2841fb2955e9 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 25 Nov 2025 16:20:44 -0500 Subject: [PATCH 31/75] removed redundant info --- modules/develop/pages/manage-topics/cloud-topics.adoc | 6 ------ 1 file changed, 6 deletions(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index 8d3e3cecf5..04f58bb9c8 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -14,12 +14,6 @@ Starting in v25.3, Redpanda provides Cloud Topics to support multi-modal streami Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud object storage (S3, ADLS,GCS, MinIO) as the primary mechanism for backing up data. This eliminates over 90% of the cost of replicating data over network links in multi-AZ or multi-region clusters. The difference in end-to-end latency is typically between 500ms-1s, which is often acceptable for many streaming workloads, and can unlock new streaming use cases that were previously not cost effective. -:tip-caption: Cost savings example - -TIP: In a cloud environment tested by Redpanda, the networking costs of data replication to run a single 100MBps workload were reduced from over $10,000 per month to $80 per month. While this does not represent 100% of total operating costs (Kafka batch sizes can affect this cost reduction slightly), at high throughputs this savings still accounts for the majority of cloud provider expense, reducing the total infrastructure cost of running Kafka workloads by an order of magnitude or more. - -:tip-caption: Tip - == Prerequisites - xref:get-started:rpk-install.adoc[] v25.3 or later. From fe7d1c7f2df945d89612273d99236adf71eea31b Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Tue, 25 Nov 2025 15:49:12 -0600 Subject: [PATCH 32/75] Apply suggestions from code review doc review feedback Co-authored-by: Michele Cyran --- docs-data/property-overrides.json | 4 ++-- modules/develop/pages/manage-topics/cloud-topics.adoc | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs-data/property-overrides.json b/docs-data/property-overrides.json index e6fa76eaab..b4d117156f 100644 --- a/docs-data/property-overrides.json +++ b/docs-data/property-overrides.json @@ -497,7 +497,7 @@ "config_scope": "cluster" }, "cloud_topics_produce_upload_interval": { - "description": "Time interval after which data is uploaded to object storage for Cloud Topics. When this time threshold is reached, Redpanda triggers an upload of buffered data to the cloud object storage backend (S3, GCS, or MinIO), regardless of whether the size or cardinality thresholds have been met.\n\nThis property works together with <> and <> to control when uploads occur. An upload is triggered when any of these three thresholds is reached.", + "description": "Time interval after which data is uploaded to object storage for Cloud Topics. When this time threshold is reached, Redpanda triggers an upload of buffered data to the object storage backend (S3, GCS, or MinIO), regardless of whether the size or cardinality thresholds have been met.\n\nThis property works together with <> and <> to control when uploads occur. An upload is triggered when any of these three thresholds is reached.", "related_topics": [ "xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics]", "xref:reference:properties/cluster-properties.adoc#cloud_topics_produce_batching_size_threshold[`cloud_topics_produce_batching_size_threshold`]", @@ -506,7 +506,7 @@ "config_scope": "cluster" }, "cloud_topics_reconciliation_interval": { - "description": "Time interval at which Redpanda reconciles data between short-term local storage and long-term cloud object storage for Cloud Topics. During reconciliation, Redpanda moves data that has been successfully persisted to object storage from local storage, freeing up local disk space while maintaining data durability in the cloud.\n\nThis reconciliation process is essential for Cloud Topics to achieve their cost-efficiency goals by minimizing the amount of expensive local storage required, while ensuring data remains accessible from durable cloud storage.", + "description": "Time interval at which Redpanda reconciles data between short-term local storage and long-term object storage for Cloud Topics. During reconciliation, Redpanda moves data that has been successfully persisted to object storage from local storage, freeing up local disk space while maintaining data durability in the cloud.\n\nThis reconciliation process is essential for Cloud Topics to achieve their cost-efficiency goals by minimizing the amount of expensive local storage required, while ensuring data remains accessible from durable cloud storage.", "related_topics": [ "xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics]", "xref:reference:properties/cluster-properties.adoc#cloud_topics_long_term_garbage_collection_interval[`cloud_topics_long_term_garbage_collection_interval`]" diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index 04f58bb9c8..150b7d017d 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -12,7 +12,7 @@ endif::[] Starting in v25.3, Redpanda provides Cloud Topics to support multi-modal streaming workloads in the most cost-effective way possible: as a configurable option on a single cluster. While standard Redpanda xref:config-topics.adoc[topics] that use local storage or Tiered Storage are ideal for latency-sensitive, low-throughput workloads (for example, for audit logs), Cloud Topics are optimized for latency-tolerant, high-throughput workloads where cloud provider infrastructure costs (especially cross-AZ networking charges) are a major consideration that can become the dominant cost driver at high throughput. These workloads can include observability streams, offline analytics, AI/ML model training data feeds, or development environments where streaming latency isn't critical. -Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud object storage (S3, ADLS,GCS, MinIO) as the primary mechanism for backing up data. This eliminates over 90% of the cost of replicating data over network links in multi-AZ or multi-region clusters. The difference in end-to-end latency is typically between 500ms-1s, which is often acceptable for many streaming workloads, and can unlock new streaming use cases that were previously not cost effective. +Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud storage (S3, ADLS,GCS, MinIO) as the primary mechanism to back up data. This eliminates over 90% of the cost of replicating data over network links in multi-AZ or multi-region clusters. The difference in end-to-end latency is typically between 500 ms - 1 s, which is often acceptable for many streaming workloads, and can unlock new streaming use cases that previously were not cost effective. == Prerequisites From b0d9b97386d4e8fd074b3b7f56ced25469515a6b Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Tue, 25 Nov 2025 16:44:19 -0600 Subject: [PATCH 33/75] Update modules/reference/attachments/redpanda-properties-v25.3.1.json Co-authored-by: Michele Cyran --- modules/reference/attachments/redpanda-properties-v25.3.7.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/reference/attachments/redpanda-properties-v25.3.7.json b/modules/reference/attachments/redpanda-properties-v25.3.7.json index adce09d41c..62246d46c9 100644 --- a/modules/reference/attachments/redpanda-properties-v25.3.7.json +++ b/modules/reference/attachments/redpanda-properties-v25.3.7.json @@ -3355,7 +3355,7 @@ "default": 250, "default_human_readable": "250 milliseconds", "defined_in": "src/v/config/configuration.cc", - "description": "Time interval after which data is uploaded to object storage for Cloud Topics. When this time threshold is reached, Redpanda triggers an upload of buffered data to the cloud object storage backend (S3, GCS, or MinIO), regardless of whether the size or cardinality thresholds have been met.\n\nThis property works together with <> and <> to control when uploads occur. An upload is triggered when any of these three thresholds is reached.", + "description": "Time interval after which data is uploaded to object storage for Cloud Topics. When this time threshold is reached, Redpanda triggers an upload of buffered data to the object storage backend (S3, GCS, or MinIO), regardless of whether the size or cardinality thresholds have been met.\n\nThis property works together with <> and <> to control when uploads occur. An upload is triggered when any of these three thresholds is reached.", "is_deprecated": false, "is_enterprise": false, "maximum": 17592186044415, From 673649aa2e5c4592a60c8dcbf5b94ba3be4a9856 Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Tue, 25 Nov 2025 16:44:45 -0600 Subject: [PATCH 34/75] Update modules/reference/partials/properties/cluster-properties.adoc Co-authored-by: Michele Cyran --- modules/reference/partials/properties/cluster-properties.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/reference/partials/properties/cluster-properties.adoc b/modules/reference/partials/properties/cluster-properties.adoc index 8b2df4e98a..6a76a7769b 100644 --- a/modules/reference/partials/properties/cluster-properties.adoc +++ b/modules/reference/partials/properties/cluster-properties.adoc @@ -1245,7 +1245,7 @@ endif::[] === cloud_topics_reconciliation_interval -Time interval at which Redpanda reconciles data between short-term local storage and long-term cloud object storage for Cloud Topics. During reconciliation, Redpanda moves data that has been successfully persisted to object storage from local storage, freeing up local disk space while maintaining data durability in the cloud. +Time interval at which Redpanda reconciles data between short-term local storage and long-term object storage for Cloud Topics. During reconciliation, Redpanda moves data that has been successfully persisted to object storage from local storage, freeing up local disk space while maintaining data durability in the cloud. This reconciliation process is essential for Cloud Topics to achieve their cost-efficiency goals by minimizing the amount of expensive local storage required, while ensuring data remains accessible from durable cloud storage. From 47528e99f05cd1c3ad6562aa5ceefcd5647a30f9 Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Tue, 25 Nov 2025 16:48:31 -0600 Subject: [PATCH 35/75] Apply suggestions from code review doc review feedback Co-authored-by: Michele Cyran --- docs-data/property-overrides.json | 2 +- .../pages/manage-topics/cloud-topics.adoc | 18 +++++++++--------- .../redpanda-properties-v25.3.7.json | 4 ++-- .../properties/cluster-properties.adoc | 2 +- .../partials/properties/topic-properties.adoc | 2 +- 5 files changed, 14 insertions(+), 14 deletions(-) diff --git a/docs-data/property-overrides.json b/docs-data/property-overrides.json index b4d117156f..054fa9f6fc 100644 --- a/docs-data/property-overrides.json +++ b/docs-data/property-overrides.json @@ -1582,7 +1582,7 @@ "category": "redpanda" }, "redpanda.cloud_topic.enabled": { - "description": "Enable this topic as a Cloud Topic. Cloud Topics are optimized for high-throughput, cost-sensitive workloads that can tolerate higher latencies compared to standard Kafka topics.\n\nIMPORTANT: You can only set this property when creating a topic. Once a topic is created as a Cloud Topic, it cannot be converted back to a standard Kafka topic. Before creating Cloud Topics, you must enable the cluster-wide <> property.", + "description": "Enable this topic as a Cloud Topic. Cloud Topics are optimized for high-throughput, cost-sensitive workloads that can tolerate higher latencies compared to standard Kafka topics.\n\nIMPORTANT: You can only set this property when creating a topic. After a topic is created as a Cloud Topic, it cannot be converted back to a standard Kafka topic. Before creating Cloud Topics, you must enable the cluster-wide <> property.", "related_topics": [ "xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics]", "xref:reference:properties/cluster-properties.adoc#cloud_topics_enabled[`cloud_topics_enabled`]" diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index 150b7d017d..a89b2f6a3c 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -1,4 +1,4 @@ -= Cloud Topics += Manage Cloud Topics :description: Cloud Topics are Redpanda topics that enable users to trade off latency for lower costs. :page-beta: true // tag::single-source[] @@ -18,11 +18,11 @@ Instead of replicating every byte across expensive network links, Cloud Topics l - xref:get-started:rpk-install.adoc[] v25.3 or later. ifndef::env-cloud[] -- xref:manage:tiered-storage.adoc#set-up-tiered-storage[Set up] and xref:manage:tiered-storage.adoc#enable-tiered-storage[enable Tiered Storage] on your Redpanda cluster. +- xref:manage:tiered-storage.adoc#set-up-tiered-storage[Enable Tiered Storage] on your Redpanda cluster. + [NOTE] ==== -When you enabled Tiered Storage at the cluster level, the default is that all new topics are Tiered Storage (or standard) topics. However, if you plan to use Cloud Topics for all new topics in a Redpanda cluster, be sure to set the following cluster-level properties to `false`: +When you enable Tiered Storage at the cluster level, by default, all new topics are Tiered Storage (or standard) topics. However, if you plan to use Cloud Topics for all new topics in a Redpanda cluster, be sure to set the following cluster-level properties to `false`: * config_ref:cloud_storage_enable_remote_write,false,properties/object-storage-properties[] * config_ref:cloud_storage_enable_remote_read,false,properties/object-storage-properties[] @@ -33,10 +33,10 @@ For details, see xref:manage:tiered-storage.adoc#enable-tiered-storage-for-a-clu ==== - xref:manage:tiered-storage.adoc#configure-object-storage[Configure object storage]. -- You must have an Enterprise license. - -To check your license status: - +- Ensure that you have an Enterprise license. ++ +To check your license status, run: ++ [,bash] ---- rpk cluster license info @@ -57,7 +57,7 @@ endif::[] == Limitations -Not all Redpanda features are available when using Cloud Topics (beta). The following features are not supported in this beta release: +The following features are not supported in this beta release of Cloud Topics: - Kafka compaction - Iceberg topics @@ -87,6 +87,6 @@ audit.analytics.may2025 OK You can make a topic a Cloud Topic only at topic creation time. -NOTE: In addition to replication, cross-AZ ingress (Producer) and egress (Consumer) traffic can also contribute substantially to cloud networking cots. When running multi-AZ clusters in general, Redpanda strongly recommends using xref:../consume-data/follower-fetching.adoc[Follower Fetching], which allows consumers to avoid crossing network zones. When possible, you can use xref:../produce-data/leader-pinning.adoc[leader pinning], which positions a topic's partition leader close to the producers, providing a similar benefit for ingress traffic. These features can add additional savings to the replication cost savings of Cloud Topics. +NOTE: In addition to replication, cross-AZ ingress (producer) and egress (consumer) traffic can also contribute substantially to cloud networking cost. When running multi-AZ clusters in general, Redpanda strongly recommends using xref:../consume-data/follower-fetching.adoc[Follower Fetching], which allows consumers to avoid crossing network zones. When possible, you can use xref:../produce-data/leader-pinning.adoc[leader pinning], which positions a topic's partition leader close to the producers, providing a similar benefit for ingress traffic. These features can add additional savings to the replication cost savings of Cloud Topics. // end::single-source[] \ No newline at end of file diff --git a/modules/reference/attachments/redpanda-properties-v25.3.7.json b/modules/reference/attachments/redpanda-properties-v25.3.7.json index 62246d46c9..c45fd78232 100644 --- a/modules/reference/attachments/redpanda-properties-v25.3.7.json +++ b/modules/reference/attachments/redpanda-properties-v25.3.7.json @@ -3381,7 +3381,7 @@ "default": 10000, "default_human_readable": "10 seconds", "defined_in": "src/v/config/configuration.cc", - "description": "Time interval at which Redpanda reconciles data between short-term local storage and long-term cloud object storage for Cloud Topics. During reconciliation, Redpanda moves data that has been successfully persisted to object storage from local storage, freeing up local disk space while maintaining data durability in the cloud.\n\nThis reconciliation process is essential for Cloud Topics to achieve their cost-efficiency goals by minimizing the amount of expensive local storage required, while ensuring data remains accessible from durable cloud storage.", + "description": "Time interval at which Redpanda reconciles data between short-term local storage and long-term object storage for Cloud Topics. During reconciliation, Redpanda moves data that has been successfully persisted to object storage from local storage, freeing up local disk space while maintaining data durability in the cloud.\n\nThis reconciliation process is essential for Cloud Topics to achieve their cost-efficiency goals by minimizing the amount of expensive local storage required, while ensuring data remains accessible from durable object storage.", "is_deprecated": false, "is_enterprise": false, "maximum": 17592186044415, @@ -11027,7 +11027,7 @@ "corresponding_cluster_property": null, "default": null, "defined_in": "src/v/kafka/protocol/topic_properties.h", - "description": "Enable this topic as a Cloud Topic. Cloud Topics are optimized for high-throughput, cost-sensitive workloads that can tolerate higher latencies compared to standard Kafka topics.\n\nIMPORTANT: You can only set this property when creating a topic. Once a topic is created as a Cloud Topic, it cannot be converted back to a standard Kafka topic. Before creating Cloud Topics, you must enable the cluster-wide <> property.", + "description": "Enable this topic as a Cloud Topic. Cloud Topics are optimized for high-throughput, cost-sensitive workloads that can tolerate higher latencies compared to standard Kafka topics.\n\nIMPORTANT: You can only set this property when creating a topic. After a topic is created as a Cloud Topic, it cannot be converted back to a standard Kafka topic. Before creating Cloud Topics, you must enable the cluster-wide <> property.", "is_deprecated": false, "is_enterprise": false, "is_topic_property": true, diff --git a/modules/reference/partials/properties/cluster-properties.adoc b/modules/reference/partials/properties/cluster-properties.adoc index 6a76a7769b..5c55a0d943 100644 --- a/modules/reference/partials/properties/cluster-properties.adoc +++ b/modules/reference/partials/properties/cluster-properties.adoc @@ -1191,7 +1191,7 @@ endif::[] === cloud_topics_produce_upload_interval -Time interval after which data is uploaded to object storage for Cloud Topics. When this time threshold is reached, Redpanda triggers an upload of buffered data to the cloud object storage backend (S3, GCS, or MinIO), regardless of whether the size or cardinality thresholds have been met. +Time interval after which data is uploaded to object storage for Cloud Topics. When this time threshold is reached, Redpanda triggers an upload of buffered data to the object storage backend (S3, GCS, or MinIO), regardless of whether the size or cardinality thresholds have been met. This property works together with <> and <> to control when uploads occur. An upload is triggered when any of these three thresholds is reached. diff --git a/modules/reference/partials/properties/topic-properties.adoc b/modules/reference/partials/properties/topic-properties.adoc index 69126c6789..28cf001524 100644 --- a/modules/reference/partials/properties/topic-properties.adoc +++ b/modules/reference/partials/properties/topic-properties.adoc @@ -949,7 +949,7 @@ endif::[] Enable this topic as a Cloud Topic. Cloud Topics are optimized for high-throughput, cost-sensitive workloads that can tolerate higher latencies compared to standard Kafka topics. -IMPORTANT: You can only set this property when creating a topic. Once a topic is created as a Cloud Topic, it cannot be converted back to a standard Kafka topic. Before creating Cloud Topics, you must enable the cluster-wide <> property. +IMPORTANT: You can only set this property when creating a topic. After a topic is created as a Cloud Topic, it cannot be converted back to a standard Kafka topic. Before creating Cloud Topics, you must enable the cluster-wide xref:reference:properties/cluster-properties.adoc#cloud_topics_enabled[`cloud_topics_enabled`] property. [cols="1s,2a"] |=== From 44980990211956a0ce2c3f6dca76dde69cf97157 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Wed, 26 Nov 2025 13:13:29 -0500 Subject: [PATCH 36/75] fixed links --- modules/develop/pages/kafka-clients.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/develop/pages/kafka-clients.adoc b/modules/develop/pages/kafka-clients.adoc index f984c9db93..978ea37dd4 100644 --- a/modules/develop/pages/kafka-clients.adoc +++ b/modules/develop/pages/kafka-clients.adoc @@ -60,7 +60,7 @@ endif::[] * HTTP Proxy (pandaproxy): Unlike other REST proxy implementations in the Kafka ecosystem, Redpanda HTTP Proxy does not support topic and ACLs CRUD through the HTTP Proxy. HTTP Proxy is designed for clients producing and consuming data that do not perform administrative functions. ifdef::env-cloud[] + -* The `delete.retention.ms` topic configuration in Kafka is not supported. Tombstone markers are not removed for topics with a `compact` xref:develop:manage-topics/config-topics.adoc#change-the-cleanup-policy[cleanup policy]. Redpanda only deletes tombstone markers when topics with a cleanup policy of `compact,delete` have reached their retention limits. +* The `delete.retention.ms` topic configuration in Kafka is not supported. Tombstone markers are not removed for topics with a `compact` xref:manage-topics/config-topics.adoc#change-the-cleanup-policy[cleanup policy]. Redpanda only deletes tombstone markers when topics with a cleanup policy of `compact,delete` have reached their retention limits. endif::[] ifndef::env-cloud[] + From 68ecd1cf92cb86dabecba41a5eb08b2fb4087ef1 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Wed, 26 Nov 2025 13:13:42 -0500 Subject: [PATCH 37/75] fix link --- .../attachments/redpanda-properties-v25.3.7.json | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/modules/reference/attachments/redpanda-properties-v25.3.7.json b/modules/reference/attachments/redpanda-properties-v25.3.7.json index c45fd78232..8789db316c 100644 --- a/modules/reference/attachments/redpanda-properties-v25.3.7.json +++ b/modules/reference/attachments/redpanda-properties-v25.3.7.json @@ -11521,8 +11521,8 @@ "xref:reference:rpk/rpk-topic/rpk-topic-describe.adoc[`rpk topic describe`]", "xref:reference:rpk/rpk-topic/rpk-topic-alter-config.adoc[`rpk topic alter-config`]", "xref:reference:properties/cluster-properties.adoc#default_topic_replication[`default_topic_replication`]", - "xref:develop:manage-topics.adoc#choose-the-replication-factor[Choose the replication factor]", - "xref:develop:manage-topics.adoc#change-the-replication-factor[Change the replication factor]", + "xref:develop:manage-topics/config-topics.adoc#choose-the-replication-factor[Choose the replication factor]", + "xref:develop:manage-topics/config-topics.adoc#change-the-replication-factor[Change the replication factor]", "xref:reference:properties/cluster-properties.adoc#default_topic_replication[default_topic_replication]" ], "type": "integer" @@ -13895,7 +13895,7 @@ "name": "write.caching", "needs_restart": false, "related_topics": [ - "xref:develop:manage-topics.adoc#configure-write-caching[Write caching]", + "xref:develop:manage-topics/config-topics.adoc#configure-write-caching[Write caching]", "xref:manage:tiered-storage.adoc[Tiered Storage]", "xref:reference:properties/cluster-properties.adoc#write_caching_default[`write_caching_default`]", "xref:cluster-properties.adoc#write_caching_default[`write_caching_default`]" @@ -13925,7 +13925,7 @@ "nullable": false, "related_topics": [ "xref:reference:properties/topic-properties.adoc#writecaching[`write.caching`]", - "xref:develop:manage-topics.adoc#configure-write-caching[Write caching]" + "xref:develop:manage-topics/config-topics.adoc#configure-write-caching[Write caching]" ], "type": "string", "visibility": "user" From e05a2ef629ea8dc5a9374cc5c09b0e62db4f1fc7 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Wed, 26 Nov 2025 13:13:54 -0500 Subject: [PATCH 38/75] fixed llinks --- modules/reference/partials/properties/cluster-properties.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/reference/partials/properties/cluster-properties.adoc b/modules/reference/partials/properties/cluster-properties.adoc index 5c55a0d943..a2ab5b07b6 100644 --- a/modules/reference/partials/properties/cluster-properties.adoc +++ b/modules/reference/partials/properties/cluster-properties.adoc @@ -19942,7 +19942,7 @@ endif::[] | * xref:reference:properties/topic-properties.adoc#writecaching[`write.caching`] -* xref:develop:manage-topics.adoc#configure-write-caching[Write caching] +* xref:develop:manage-topics/config-topics.adoc#configure-write-caching[Write caching] |=== From 2f9ade6fa951e3082e1871099296a76b86d07761 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Wed, 26 Nov 2025 13:14:05 -0500 Subject: [PATCH 39/75] fixed links --- modules/reference/partials/properties/topic-properties.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/reference/partials/properties/topic-properties.adoc b/modules/reference/partials/properties/topic-properties.adoc index 28cf001524..8887c64c8b 100644 --- a/modules/reference/partials/properties/topic-properties.adoc +++ b/modules/reference/partials/properties/topic-properties.adoc @@ -1715,9 +1715,9 @@ endif::[] * xref:reference:properties/cluster-properties.adoc#default_topic_replication[`default_topic_replication`] -* xref:develop:manage-topics.adoc#choose-the-replication-factor[Choose the replication factor] +* xref:develop:manage-topics/config-topics.adoc#choose-the-replication-factor[Choose the replication factor] -* xref:develop:manage-topics.adoc#change-the-replication-factor[Change the replication factor] +* xref:develop:manage-topics/config-topics.adoc#change-the-replication-factor[Change the replication factor] * xref:reference:properties/cluster-properties.adoc#default_topic_replication[default_topic_replication] @@ -2111,7 +2111,7 @@ endif::[] | Related topics | -* xref:develop:manage-topics.adoc#configure-write-caching[Write caching] +* xref:develop:manage-topics/config-topics.adoc#configure-write-caching[Write caching] * xref:manage:tiered-storage.adoc[Tiered Storage] From bdbf49eed57aeecdfccfb34ee31da4d8b23cba39 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Wed, 26 Nov 2025 14:35:03 -0500 Subject: [PATCH 40/75] fix broken link --- modules/develop/pages/kafka-clients.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/develop/pages/kafka-clients.adoc b/modules/develop/pages/kafka-clients.adoc index 978ea37dd4..f984c9db93 100644 --- a/modules/develop/pages/kafka-clients.adoc +++ b/modules/develop/pages/kafka-clients.adoc @@ -60,7 +60,7 @@ endif::[] * HTTP Proxy (pandaproxy): Unlike other REST proxy implementations in the Kafka ecosystem, Redpanda HTTP Proxy does not support topic and ACLs CRUD through the HTTP Proxy. HTTP Proxy is designed for clients producing and consuming data that do not perform administrative functions. ifdef::env-cloud[] + -* The `delete.retention.ms` topic configuration in Kafka is not supported. Tombstone markers are not removed for topics with a `compact` xref:manage-topics/config-topics.adoc#change-the-cleanup-policy[cleanup policy]. Redpanda only deletes tombstone markers when topics with a cleanup policy of `compact,delete` have reached their retention limits. +* The `delete.retention.ms` topic configuration in Kafka is not supported. Tombstone markers are not removed for topics with a `compact` xref:develop:manage-topics/config-topics.adoc#change-the-cleanup-policy[cleanup policy]. Redpanda only deletes tombstone markers when topics with a cleanup policy of `compact,delete` have reached their retention limits. endif::[] ifndef::env-cloud[] + From 0507080969d59bf0cc36daf58d7021239fd9fd20 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Wed, 26 Nov 2025 15:12:41 -0500 Subject: [PATCH 41/75] removed link --- modules/develop/pages/kafka-clients.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/develop/pages/kafka-clients.adoc b/modules/develop/pages/kafka-clients.adoc index f984c9db93..fd115b185e 100644 --- a/modules/develop/pages/kafka-clients.adoc +++ b/modules/develop/pages/kafka-clients.adoc @@ -60,7 +60,7 @@ endif::[] * HTTP Proxy (pandaproxy): Unlike other REST proxy implementations in the Kafka ecosystem, Redpanda HTTP Proxy does not support topic and ACLs CRUD through the HTTP Proxy. HTTP Proxy is designed for clients producing and consuming data that do not perform administrative functions. ifdef::env-cloud[] + -* The `delete.retention.ms` topic configuration in Kafka is not supported. Tombstone markers are not removed for topics with a `compact` xref:develop:manage-topics/config-topics.adoc#change-the-cleanup-policy[cleanup policy]. Redpanda only deletes tombstone markers when topics with a cleanup policy of `compact,delete` have reached their retention limits. +* The `delete.retention.ms` topic configuration in Kafka is not supported. Tombstone markers are not removed for topics with a `compact` cleanup policy. Redpanda only deletes tombstone markers when topics with a cleanup policy of `compact,delete` have reached their retention limits. endif::[] ifndef::env-cloud[] + From 9222987311902a3d03c0c58a90e833133f8d66d3 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Wed, 3 Dec 2025 11:21:12 -0500 Subject: [PATCH 42/75] review feedback from Matt --- modules/develop/pages/manage-topics/cloud-topics.adoc | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index a89b2f6a3c..1aaea62bc3 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -14,6 +14,8 @@ Starting in v25.3, Redpanda provides Cloud Topics to support multi-modal streami Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud storage (S3, ADLS,GCS, MinIO) as the primary mechanism to back up data. This eliminates over 90% of the cost of replicating data over network links in multi-AZ or multi-region clusters. The difference in end-to-end latency is typically between 500 ms - 1 s, which is often acceptable for many streaming workloads, and can unlock new streaming use cases that previously were not cost effective. +Also, with Cloud Topics data from the client is not acknowledged until it is uploaded to cloud storage. This maintains durability in the face of infrastructure failures, but results in an increase in both produce latency and end-to-end-latency, driven by both batching of produced data and the inherent latency of the underlying object store. Latency varies somewhat depending on throughput (higher throughputs will achieve somewhat lower latency), you should generally expect latencies in the range of 250 ms - 1 s. + == Prerequisites - xref:get-started:rpk-install.adoc[] v25.3 or later. @@ -87,6 +89,8 @@ audit.analytics.may2025 OK You can make a topic a Cloud Topic only at topic creation time. -NOTE: In addition to replication, cross-AZ ingress (producer) and egress (consumer) traffic can also contribute substantially to cloud networking cost. When running multi-AZ clusters in general, Redpanda strongly recommends using xref:../consume-data/follower-fetching.adoc[Follower Fetching], which allows consumers to avoid crossing network zones. When possible, you can use xref:../produce-data/leader-pinning.adoc[leader pinning], which positions a topic's partition leader close to the producers, providing a similar benefit for ingress traffic. These features can add additional savings to the replication cost savings of Cloud Topics. +TIP: Given the architecture and latency profile of Cloud Topics, a producer configuration of `acks=all` is always recommended, as other configurations (for example, `acks=0` or `acks=1`) will not yield a lower produce latency. + +In addition to replication, cross-AZ ingress (producer) and egress (consumer) traffic can also contribute substantially to cloud networking cost. When running multi-AZ clusters in general, Redpanda strongly recommends using xref:../consume-data/follower-fetching.adoc[Follower Fetching], which allows consumers to avoid crossing network zones. When possible, you can use xref:../produce-data/leader-pinning.adoc[leader pinning], which positions a topic's partition leader close to the producers, providing a similar benefit for ingress traffic. These features can add additional savings to the replication cost savings of Cloud Topics. // end::single-source[] \ No newline at end of file From d3d0562e1c62db50a21a49ec82afce0fcaa24d41 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Wed, 3 Dec 2025 12:37:27 -0500 Subject: [PATCH 43/75] copyedit --- modules/develop/pages/manage-topics/cloud-topics.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index 1aaea62bc3..6f882d2d32 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -12,9 +12,9 @@ endif::[] Starting in v25.3, Redpanda provides Cloud Topics to support multi-modal streaming workloads in the most cost-effective way possible: as a configurable option on a single cluster. While standard Redpanda xref:config-topics.adoc[topics] that use local storage or Tiered Storage are ideal for latency-sensitive, low-throughput workloads (for example, for audit logs), Cloud Topics are optimized for latency-tolerant, high-throughput workloads where cloud provider infrastructure costs (especially cross-AZ networking charges) are a major consideration that can become the dominant cost driver at high throughput. These workloads can include observability streams, offline analytics, AI/ML model training data feeds, or development environments where streaming latency isn't critical. -Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud storage (S3, ADLS,GCS, MinIO) as the primary mechanism to back up data. This eliminates over 90% of the cost of replicating data over network links in multi-AZ or multi-region clusters. The difference in end-to-end latency is typically between 500 ms - 1 s, which is often acceptable for many streaming workloads, and can unlock new streaming use cases that previously were not cost effective. +Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud storage (S3, ADLS, GCS, MinIO) as the primary mechanism to back up data. This eliminates over 90% of the cost of replicating data over network links in multi-AZ or multi-region clusters. The difference in end-to-end latency is typically between 500 ms - 1 s, which is often acceptable for many streaming workloads, and can unlock new streaming use cases that previously were not cost effective. -Also, with Cloud Topics data from the client is not acknowledged until it is uploaded to cloud storage. This maintains durability in the face of infrastructure failures, but results in an increase in both produce latency and end-to-end-latency, driven by both batching of produced data and the inherent latency of the underlying object store. Latency varies somewhat depending on throughput (higher throughputs will achieve somewhat lower latency), you should generally expect latencies in the range of 250 ms - 1 s. +Also, with Cloud Topics data from the client is not acknowledged until it is uploaded to cloud storage. This maintains durability in the face of infrastructure failures, but results in an increase in both produce latency and end-to-end latency, driven by both batching of produced data and the inherent latency of the underlying object store. Latency varies somewhat depending on throughput (higher throughputs will achieve somewhat lower latency). You should generally expect latencies in the range of 250 ms - 1 s. == Prerequisites @@ -89,7 +89,7 @@ audit.analytics.may2025 OK You can make a topic a Cloud Topic only at topic creation time. -TIP: Given the architecture and latency profile of Cloud Topics, a producer configuration of `acks=all` is always recommended, as other configurations (for example, `acks=0` or `acks=1`) will not yield a lower produce latency. +TIP: Given the architecture and latency profile of Cloud Topics, a producer configuration of `acks=all` is always recommended, as other configurations (for example, `acks=0` or `acks=1`) will not yield a lower produce latency. In addition to replication, cross-AZ ingress (producer) and egress (consumer) traffic can also contribute substantially to cloud networking cost. When running multi-AZ clusters in general, Redpanda strongly recommends using xref:../consume-data/follower-fetching.adoc[Follower Fetching], which allows consumers to avoid crossing network zones. When possible, you can use xref:../produce-data/leader-pinning.adoc[leader pinning], which positions a topic's partition leader close to the producers, providing a similar benefit for ingress traffic. These features can add additional savings to the replication cost savings of Cloud Topics. From 6ce969051045d4088f51251fe48f62f429a6eb6a Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Mon, 8 Dec 2025 16:09:04 -0500 Subject: [PATCH 44/75] Removed per Matt Shumpert --- modules/develop/pages/manage-topics/cloud-topics.adoc | 2 -- 1 file changed, 2 deletions(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index 6f882d2d32..8bed459815 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -89,8 +89,6 @@ audit.analytics.may2025 OK You can make a topic a Cloud Topic only at topic creation time. -TIP: Given the architecture and latency profile of Cloud Topics, a producer configuration of `acks=all` is always recommended, as other configurations (for example, `acks=0` or `acks=1`) will not yield a lower produce latency. - In addition to replication, cross-AZ ingress (producer) and egress (consumer) traffic can also contribute substantially to cloud networking cost. When running multi-AZ clusters in general, Redpanda strongly recommends using xref:../consume-data/follower-fetching.adoc[Follower Fetching], which allows consumers to avoid crossing network zones. When possible, you can use xref:../produce-data/leader-pinning.adoc[leader pinning], which positions a topic's partition leader close to the producers, providing a similar benefit for ingress traffic. These features can add additional savings to the replication cost savings of Cloud Topics. // end::single-source[] \ No newline at end of file From 7d2fe1889005135176e3645638b6742c5af26b8b Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Mon, 23 Feb 2026 16:57:20 -0500 Subject: [PATCH 45/75] Fix broken xref in tristate-behavior-change partial Remove hardcoded 25.3@ version qualifier from the xref so it resolves correctly in the 26.1 preview build. The [[behavior-changes]] anchor exists in the 26.1 release notes, making the version prefix unnecessary. Co-Authored-By: Claude Sonnet 4.6 --- modules/shared/partials/tristate-behavior-change-25-3.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/shared/partials/tristate-behavior-change-25-3.adoc b/modules/shared/partials/tristate-behavior-change-25-3.adoc index e79d95b59a..064f496d63 100644 --- a/modules/shared/partials/tristate-behavior-change-25-3.adoc +++ b/modules/shared/partials/tristate-behavior-change-25-3.adoc @@ -1,5 +1,5 @@ Starting in Redpanda v25.3, several topic properties support enhanced tristate behavior. Properties like `retention.ms`, `retention.bytes`, `segment.ms`, and others now distinguish between zero values (immediate eligibility for cleanup/compaction) and negative values (disable the feature entirely). Previously, zero and negative values were treated the same way. ifndef::env-cloud[] -For the complete list of affected properties and detailed information, see xref:25.3@get-started:release-notes/redpanda.adoc#behavior-changes[Redpanda v25.3 behavior changes]. +For the complete list of affected properties and detailed information, see xref:get-started:release-notes/redpanda.adoc#behavior-changes[Redpanda v25.3 behavior changes]. endif::[] Review your topic configurations if you currently use zero values for these properties. \ No newline at end of file From f8f8751e53d00dba86cd0a80f8c819fcd30d5134 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Mon, 23 Feb 2026 17:10:42 -0500 Subject: [PATCH 46/75] Add explicit anchor to Configure write caching section The xref from shadowing/overview.adoc targets #configure-write-caching but the section had no explicit anchor, causing Asciidoctor to fail on the auto-generated underscore form. Adding [[configure-write-caching]] fixes the broken xref from cloud-docs. Co-Authored-By: Claude Sonnet 4.6 --- modules/develop/pages/manage-topics/config-topics.adoc | 1 + 1 file changed, 1 insertion(+) diff --git a/modules/develop/pages/manage-topics/config-topics.adoc b/modules/develop/pages/manage-topics/config-topics.adoc index 91515616c9..f4b31c422f 100644 --- a/modules/develop/pages/manage-topics/config-topics.adoc +++ b/modules/develop/pages/manage-topics/config-topics.adoc @@ -114,6 +114,7 @@ For details on compaction in Redpanda, see xref:manage:cluster-maintenance/compa endif::[] +[[configure-write-caching]] === Configure write caching Write caching is a relaxed mode of xref:develop:produce-data/configure-producers.adoc#acksall[`acks=all`] that provides better performance at the expense of durability. It acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to be written to disk. This provides lower latency while still ensuring that a majority of brokers acknowledge the write. From 29c381b2247022f21b87b7a2b36adb44989584d2 Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Mon, 23 Feb 2026 18:39:32 -0600 Subject: [PATCH 47/75] Apply suggestion from @Feediver1 feedback from eng --- modules/develop/pages/manage-topics/cloud-topics.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index 8bed459815..080aeaa642 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -82,7 +82,7 @@ Once enabled, create the Cloud Topic: [,bash] ---- -rpk topic create -c cloud_topics_enabled=true +rpk topic create -c redpanda.cloud_topic.enabled=true TOPIC STATUS audit.analytics.may2025 OK ---- From 63b6084beded8d7be7ce9ea3efba34721b423016 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 24 Feb 2026 11:37:06 -0500 Subject: [PATCH 48/75] Fix flaky Doc Detective test for Topics nav link After clicking 'View' on a broker detail, the Console may render the sidebar in a collapsed/transitional state, causing the nav-link-Topics element to be absent from the DOM (the NavLink component omits the data-testid span when isCollapsed=true). Adding a goTo to return to the overview page before finding the Topics nav link ensures a consistent sidebar state and prevents the timeout failure. The same step was also failing on main, confirming a pre-existing issue. Co-Authored-By: Claude Sonnet 4.6 --- modules/get-started/pages/quick-start.adoc | 1 + 1 file changed, 1 insertion(+) diff --git a/modules/get-started/pages/quick-start.adoc b/modules/get-started/pages/quick-start.adoc index ebbab608be..ce37ac1f99 100644 --- a/modules/get-started/pages/quick-start.adoc +++ b/modules/get-started/pages/quick-start.adoc @@ -205,6 +205,7 @@ You can filter messages in topics using several methods: Suppose you're asked to find all transactions related to the `.edu` domain. You can use a JavaScript filter to display only messages that include email addresses in that domain. . In the menu, click *Topics*. +// (step {"goTo": "http://localhost:8080/overview"}) // (step {"find": {"selector": "[data-testid='nav-link-Topics']", "elementText": "Topics", "click": true, "timeout": 10000}}) . Click the *transactions* topic. When the topic's page opens, the *Messages* tab is selected by default. // (step {"goTo": "http://localhost:8080/topics/transactions"}) From 2a4f1ab4cb3d3190722976931b7617635ada0174 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 24 Feb 2026 12:16:39 -0500 Subject: [PATCH 49/75] fix(tests): reset to overview before Topics nav click in audit logs section The upload-license page uses a special layout that can hide or prevent interaction with the standard sidebar nav. Added a goTo overview step before finding nav-link-Topics to ensure a stable sidebar state, matching the same fix applied earlier in the View topics section. Co-Authored-By: Claude Sonnet 4.6 --- modules/get-started/pages/quick-start.adoc | 1 + 1 file changed, 1 insertion(+) diff --git a/modules/get-started/pages/quick-start.adoc b/modules/get-started/pages/quick-start.adoc index ce37ac1f99..7bced71c8c 100644 --- a/modules/get-started/pages/quick-start.adoc +++ b/modules/get-started/pages/quick-start.adoc @@ -404,6 +404,7 @@ See also: Audit logs provide a record of all user actions. You can use these logs to track changes, troubleshoot issues, and maintain compliance with your organization's security policies. . In the menu, click *Topics*. +// (step {"goTo": "http://localhost:8080/overview"}) // (step {"find": { "selector": "[data-testid='nav-link-Topics']", "click": true, "timeout": 15000}}) // (step {"goTo": "http://localhost:8080/topics"}) . Click the *Show internal topics* checkbox. From 1b98b6f7b6bfecf75d289629ee5d512d2ac4f79e Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 24 Feb 2026 14:23:30 -0500 Subject: [PATCH 50/75] fix(tests): use nav-link-icon-* selectors instead of nav-link-* for sidebar nav The nav-link-{title} data-testid is only rendered on the text span when the sidebar is NOT collapsed, but nav-link-icon-{title} is always rendered on the icon element regardless of sidebar state. The icon is inside the anchor link so clicking it still triggers navigation. This fixes consistent CI failures where the sidebar was in a collapsed state (isNavCollapsed=true in localStorage) after earlier navigations, causing the text span to be absent from the DOM even after a goTo reset. Co-Authored-By: Claude Sonnet 4.6 --- modules/get-started/pages/quick-start.adoc | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/modules/get-started/pages/quick-start.adoc b/modules/get-started/pages/quick-start.adoc index 7bced71c8c..a764ac9c0f 100644 --- a/modules/get-started/pages/quick-start.adoc +++ b/modules/get-started/pages/quick-start.adoc @@ -206,7 +206,7 @@ Suppose you're asked to find all transactions related to the `.edu` domain. You . In the menu, click *Topics*. // (step {"goTo": "http://localhost:8080/overview"}) -// (step {"find": {"selector": "[data-testid='nav-link-Topics']", "elementText": "Topics", "click": true, "timeout": 10000}}) +// (step {"find": {"selector": "[data-testid='nav-link-icon-Topics']", "click": true, "timeout": 10000}}) . Click the *transactions* topic. When the topic's page opens, the *Messages* tab is selected by default. // (step {"goTo": "http://localhost:8080/topics/transactions"}) . Click *Add filter* > *JavaScript Filter*. @@ -242,7 +242,7 @@ See also: xref:console:ui/programmable-push-filters.adoc[]. On the *Schema Registry* page, you see an overview of your schemas. You can create, manage, and inspect your schemas without leaving Redpanda Console. -// (step {"find": {"selector": "[data-testid='nav-link-Schema Registry']", "elementText": "Schema Registry", "click": true, "timeout": 10000}}) +// (step {"find": {"selector": "[data-testid='nav-link-icon-Schema Registry']", "click": true, "timeout": 10000}}) // (step {"goTo": "http://localhost:8080/schema-registry/subjects/transactions?version=latest"}) // (step {"screenshot": {"path": "../../modules/console/images/schema-reg.png", "maxVariation": 0.10, "overwrite": "aboveVariation"}}) @@ -269,7 +269,7 @@ On the *Security* page, you can: Suppose you're onboarding a new team member who needs access to specific topics. You can use Redpanda Console to ensure users have the right permissions to perform their tasks. . In the menu, click *Security*. -// (step {"find": {"selector": "[data-testid='nav-link-Security']", "elementText": "Security", "click": true, "timeout": 10000}}) +// (step {"find": {"selector": "[data-testid='nav-link-icon-Security']", "click": true, "timeout": 10000}}) // (step {"goTo": "http://localhost:8080/security/users"}) . On the *Users* tab, click *Create user*. // (step {"find": {"selector": "[data-testid='create-user-button']", "elementText": "Create user", "click": true, "timeout": 100000}}) @@ -383,7 +383,7 @@ Data transforms let you run common data streaming tasks on the Redpanda broker, This quickstart deployment comes with one transform function called `regex` running in your cluster. Its job is to find records in the `logins` topic that contain email addresses with the `.edu` domain and add those to a new topic called `edu-filtered-domains`. In the menu, click *Transforms*. -// (step {"find": { "selector": "[data-testid='nav-link-Transforms']", "elementText": "Transforms", "click": true, "timeout": 15000}}) +// (step {"find": { "selector": "[data-testid='nav-link-icon-Transforms']", "click": true, "timeout": 15000}}) On the *Transforms* page, you see your transform. You can use Redpanda Console to manage and monitor your transforms. // (step {"screenshot": {"path": "../../modules/console/images/transforms.png", "maxVariation": 0.10, "overwrite": "aboveVariation"}}) @@ -405,7 +405,7 @@ Audit logs provide a record of all user actions. You can use these logs to track . In the menu, click *Topics*. // (step {"goTo": "http://localhost:8080/overview"}) -// (step {"find": { "selector": "[data-testid='nav-link-Topics']", "click": true, "timeout": 15000}}) +// (step {"find": { "selector": "[data-testid='nav-link-icon-Topics']", "click": true, "timeout": 15000}}) // (step {"goTo": "http://localhost:8080/topics"}) . Click the *Show internal topics* checkbox. // (step {"find": { "selector": "[data-testid='show-internal-topics-checkbox']", "elementText": "Show internal topics", "click": true, "timeout": 15000}}) From e91bd0cdb6d7a59acccbc8d8878f0360fcde9f72 Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Tue, 24 Feb 2026 16:47:07 -0600 Subject: [PATCH 51/75] Apply suggestion from @Feediver1 --- modules/develop/pages/manage-topics/cloud-topics.adoc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index 080aeaa642..ed32fec8ad 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -80,6 +80,8 @@ Successfully updated configuration. New configuration version is 3. Once enabled, create the Cloud Topic: +NOTE: This configuration update requires a restart. + [,bash] ---- rpk topic create -c redpanda.cloud_topic.enabled=true From cd4782e366daf9df6ba007bc0705af4c998ffb6b Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Thu, 5 Mar 2026 13:22:20 -0500 Subject: [PATCH 52/75] updates for 26.1 --- modules/develop/pages/manage-topics/cloud-topics.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index ed32fec8ad..be9a485c0a 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -10,7 +10,7 @@ include::shared:partial$enterprise-license.adoc[] ==== endif::[] -Starting in v25.3, Redpanda provides Cloud Topics to support multi-modal streaming workloads in the most cost-effective way possible: as a configurable option on a single cluster. While standard Redpanda xref:config-topics.adoc[topics] that use local storage or Tiered Storage are ideal for latency-sensitive, low-throughput workloads (for example, for audit logs), Cloud Topics are optimized for latency-tolerant, high-throughput workloads where cloud provider infrastructure costs (especially cross-AZ networking charges) are a major consideration that can become the dominant cost driver at high throughput. These workloads can include observability streams, offline analytics, AI/ML model training data feeds, or development environments where streaming latency isn't critical. +Starting in v26.1, Redpanda provides Cloud Topics to support multi-modal streaming workloads in the most cost-effective way possible: as a configurable option on a single cluster. While standard Redpanda xref:config-topics.adoc[topics] that use local storage or Tiered Storage are ideal for latency-sensitive, low-throughput workloads (for example, for audit logs or analytics), Cloud Topics are optimized for latency-tolerant, high-throughput workloads where cloud provider infrastructure costs (especially cross-AZ networking charges) are a major consideration that can become the dominant cost driver at high throughput. These workloads can include observability streams, offline analytics, AI/ML model training data feeds, or development environments that have flexible latency requirements. Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud storage (S3, ADLS, GCS, MinIO) as the primary mechanism to back up data. This eliminates over 90% of the cost of replicating data over network links in multi-AZ or multi-region clusters. The difference in end-to-end latency is typically between 500 ms - 1 s, which is often acceptable for many streaming workloads, and can unlock new streaming use cases that previously were not cost effective. @@ -84,7 +84,7 @@ NOTE: This configuration update requires a restart. [,bash] ---- -rpk topic create -c redpanda.cloud_topic.enabled=true +rpk topic create -c redpanda.storage.mode=cloud TOPIC STATUS audit.analytics.may2025 OK ---- From 98cc85261c537a02858a2afeca8b1b38ccdef57a Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Thu, 5 Mar 2026 19:46:05 -0500 Subject: [PATCH 53/75] docs: add learning objectives to cloud-topics.adoc Co-Authored-By: Claude Sonnet 4.6 --- modules/develop/pages/manage-topics/cloud-topics.adoc | 3 +++ 1 file changed, 3 insertions(+) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index be9a485c0a..322c21a8cf 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -1,6 +1,9 @@ = Manage Cloud Topics :description: Cloud Topics are Redpanda topics that enable users to trade off latency for lower costs. :page-beta: true +:learning-objective-1: Describe the latency and cost trade-offs of Cloud Topics compared to standard Redpanda topics +:learning-objective-2: Create a Cloud Topic using rpk after enabling Cloud Topics on your cluster +:learning-objective-3: Identify Cloud Topics limitations and configurations that reduce cross-AZ networking costs // tag::single-source[] ifndef::env-cloud[] From 6be2d03c0e10938064c47955dfdd7afe1448677c Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Thu, 5 Mar 2026 21:16:20 -0500 Subject: [PATCH 54/75] fix: remove conflicting page alias from manage-topics/config-topics.adoc The alias develop:config-topics.adoc conflicts with the existing page at modules/develop/pages/config-topics.adoc, causing a fatal Antora build error. Co-Authored-By: Claude Sonnet 4.6 --- modules/develop/pages/manage-topics/config-topics.adoc | 1 - 1 file changed, 1 deletion(-) diff --git a/modules/develop/pages/manage-topics/config-topics.adoc b/modules/develop/pages/manage-topics/config-topics.adoc index f4b31c422f..d258d78021 100644 --- a/modules/develop/pages/manage-topics/config-topics.adoc +++ b/modules/develop/pages/manage-topics/config-topics.adoc @@ -1,7 +1,6 @@ = Manage Topics :page-categories: Clients, Development :description: Learn how to create topics, update topic configurations, and delete topics or records. -:page-aliases: develop:config-topics.adoc // tag::single-source[] include::develop:partial$topic-defaults.adoc[] From d1280715c7c7454c48ab62dd9db51f387a455fc0 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Thu, 5 Mar 2026 21:25:50 -0500 Subject: [PATCH 55/75] chore: retrigger build From 97dc05a954b8edf530d29539fa605a7aa59e58e4 Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Tue, 10 Mar 2026 14:09:02 -0500 Subject: [PATCH 56/75] Apply suggestion from Noah --- docs-data/property-overrides.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs-data/property-overrides.json b/docs-data/property-overrides.json index 054fa9f6fc..2831b0270e 100644 --- a/docs-data/property-overrides.json +++ b/docs-data/property-overrides.json @@ -506,7 +506,7 @@ "config_scope": "cluster" }, "cloud_topics_reconciliation_interval": { - "description": "Time interval at which Redpanda reconciles data between short-term local storage and long-term object storage for Cloud Topics. During reconciliation, Redpanda moves data that has been successfully persisted to object storage from local storage, freeing up local disk space while maintaining data durability in the cloud.\n\nThis reconciliation process is essential for Cloud Topics to achieve their cost-efficiency goals by minimizing the amount of expensive local storage required, while ensuring data remains accessible from durable cloud storage.", + "description": "Time interval at which Redpanda reconciles data between short-term local storage and long-term object storage for Cloud Topics. During this reconciliation process, Redpanda optimizes the storage layout of data in short-term storage to improve the cost and performance associated with accessing data. After the reconciliation process has moved data into long-term storage, the data in short-term storage is subject to removal by a garbage collection process." "related_topics": [ "xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics]", "xref:reference:properties/cluster-properties.adoc#cloud_topics_long_term_garbage_collection_interval[`cloud_topics_long_term_garbage_collection_interval`]" From 1a57c474d324f02954b848566a8155dedcd9fd0a Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Tue, 10 Mar 2026 14:14:33 -0500 Subject: [PATCH 57/75] Apply suggestion from Noah --- modules/develop/pages/manage-topics/cloud-topics.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index 322c21a8cf..ce42c854fb 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -13,7 +13,7 @@ include::shared:partial$enterprise-license.adoc[] ==== endif::[] -Starting in v26.1, Redpanda provides Cloud Topics to support multi-modal streaming workloads in the most cost-effective way possible: as a configurable option on a single cluster. While standard Redpanda xref:config-topics.adoc[topics] that use local storage or Tiered Storage are ideal for latency-sensitive, low-throughput workloads (for example, for audit logs or analytics), Cloud Topics are optimized for latency-tolerant, high-throughput workloads where cloud provider infrastructure costs (especially cross-AZ networking charges) are a major consideration that can become the dominant cost driver at high throughput. These workloads can include observability streams, offline analytics, AI/ML model training data feeds, or development environments that have flexible latency requirements. +Starting in v26.1, Redpanda provides Cloud Topics to support multi-modal streaming workloads in the most cost-effective way possible: as a configurable option on a single cluster. While standard Redpanda xref:config-topics.adoc[topics] that use local storage or Tiered Storage are ideal for latency-tolerant workloads (for example, for audit logs or analytics), Cloud Topics are optimized for latency-tolerant, high-throughput workloads where cross-AZ networking charges are a major consideration that can become the dominant cost driver at high throughput. These workloads can include observability streams, offline analytics, AI/ML model training data feeds, or development environments that have flexible latency requirements. Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud storage (S3, ADLS, GCS, MinIO) as the primary mechanism to back up data. This eliminates over 90% of the cost of replicating data over network links in multi-AZ or multi-region clusters. The difference in end-to-end latency is typically between 500 ms - 1 s, which is often acceptable for many streaming workloads, and can unlock new streaming use cases that previously were not cost effective. From d1a52f8e08b3474b47a3c5b0068ab49081fd436a Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Tue, 10 Mar 2026 14:17:18 -0500 Subject: [PATCH 58/75] Apply suggestion from Noah --- modules/develop/pages/manage-topics/cloud-topics.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index ce42c854fb..d6f635f451 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -15,7 +15,7 @@ endif::[] Starting in v26.1, Redpanda provides Cloud Topics to support multi-modal streaming workloads in the most cost-effective way possible: as a configurable option on a single cluster. While standard Redpanda xref:config-topics.adoc[topics] that use local storage or Tiered Storage are ideal for latency-tolerant workloads (for example, for audit logs or analytics), Cloud Topics are optimized for latency-tolerant, high-throughput workloads where cross-AZ networking charges are a major consideration that can become the dominant cost driver at high throughput. These workloads can include observability streams, offline analytics, AI/ML model training data feeds, or development environments that have flexible latency requirements. -Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud storage (S3, ADLS, GCS, MinIO) as the primary mechanism to back up data. This eliminates over 90% of the cost of replicating data over network links in multi-AZ or multi-region clusters. The difference in end-to-end latency is typically between 500 ms - 1 s, which is often acceptable for many streaming workloads, and can unlock new streaming use cases that previously were not cost effective. +Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud storage (S3, ADLS, GCS, MinIO) as the primary mechanism to back up data. This eliminates over 90% of the cost of replicating data over network links in multi-AZ clusters. The difference in end-to-end latency is typically between 500 ms - 1 s, which is often acceptable for many streaming workloads, and can unlock new streaming use cases that previously were not cost effective. Also, with Cloud Topics data from the client is not acknowledged until it is uploaded to cloud storage. This maintains durability in the face of infrastructure failures, but results in an increase in both produce latency and end-to-end latency, driven by both batching of produced data and the inherent latency of the underlying object store. Latency varies somewhat depending on throughput (higher throughputs will achieve somewhat lower latency). You should generally expect latencies in the range of 250 ms - 1 s. From ca17951a41e37d036c5d1fa361e00495d9a918b0 Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Tue, 10 Mar 2026 14:19:34 -0500 Subject: [PATCH 59/75] Apply suggestion from Noah --- modules/develop/pages/manage-topics/cloud-topics.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index d6f635f451..8c371ef327 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -17,7 +17,7 @@ Starting in v26.1, Redpanda provides Cloud Topics to support multi-modal streami Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive Cloud storage (S3, ADLS, GCS, MinIO) as the primary mechanism to back up data. This eliminates over 90% of the cost of replicating data over network links in multi-AZ clusters. The difference in end-to-end latency is typically between 500 ms - 1 s, which is often acceptable for many streaming workloads, and can unlock new streaming use cases that previously were not cost effective. -Also, with Cloud Topics data from the client is not acknowledged until it is uploaded to cloud storage. This maintains durability in the face of infrastructure failures, but results in an increase in both produce latency and end-to-end latency, driven by both batching of produced data and the inherent latency of the underlying object store. Latency varies somewhat depending on throughput (higher throughputs will achieve somewhat lower latency). You should generally expect latencies in the range of 250 ms - 1 s. +With Cloud Topics, data from the client is not acknowledged until it is uploaded to cloud storage. This maintains durability in the face of infrastructure failures, but results in an increase in both produce latency and end-to-end latency, driven by both batching of produced data and the inherent latency of the underlying object store. You should generally expect latencies in the range of 250 ms - 1 s. == Prerequisites From b5b504300ffb5fab2894e9623c220085dd867c58 Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Tue, 10 Mar 2026 14:32:23 -0500 Subject: [PATCH 60/75] Apply suggestion from Noah --- modules/develop/pages/manage-topics/cloud-topics.adoc | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index 8c371ef327..765c77d60b 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -62,12 +62,7 @@ endif::[] == Limitations -The following features are not supported in this beta release of Cloud Topics: - -- Kafka compaction -- Iceberg topics -- Topic recovery -- Remote Read Replicas +The Remote Read Replicas feature is not supported for Cloud Topics. IMPORTANT: You should expect to throw away any cluster running Cloud Topics (beta) after it becomes generally available (GA). Redpanda may be unable to continue using Cloud Topics created with the beta version, and you may be unable to upgrade to future versions. Also, any topic specified to be a Cloud Topic cannot be subsequently converted back to a standard Redpanda topic that uses local or Tiered Storage. From cb4fd246bdbaa678f1b24f54f6d13c31458a7993 Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Tue, 10 Mar 2026 14:35:40 -0500 Subject: [PATCH 61/75] Apply suggestion from Noah --- modules/develop/pages/manage-topics/cloud-topics.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index 765c77d60b..6cfdfd15ea 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -64,7 +64,7 @@ endif::[] The Remote Read Replicas feature is not supported for Cloud Topics. -IMPORTANT: You should expect to throw away any cluster running Cloud Topics (beta) after it becomes generally available (GA). Redpanda may be unable to continue using Cloud Topics created with the beta version, and you may be unable to upgrade to future versions. Also, any topic specified to be a Cloud Topic cannot be subsequently converted back to a standard Redpanda topic that uses local or Tiered Storage. +IMPORTANT: Any topic specified to be a Cloud Topic cannot be subsequently converted back to a standard Redpanda topic that uses local or Tiered Storage. == Enable Cloud Topics for your cluster From 1329b5c717d11ad4a3f6700cc37821609c41cbd7 Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Tue, 10 Mar 2026 14:39:36 -0500 Subject: [PATCH 62/75] Apply suggestion from @Feediver1 --- modules/develop/pages/manage-topics/cloud-topics.adoc | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index 6cfdfd15ea..9575a3158f 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -68,16 +68,13 @@ IMPORTANT: Any topic specified to be a Cloud Topic cannot be subsequently conver == Enable Cloud Topics for your cluster -You must first enable Cloud Topics for your cluster: +To enable Cloud Topics: [,bash] ---- -rpk cluster config set unstable_beta_feature_cloud_topics_enabled=true -Successfully updated configuration. New configuration version is 3. +rpk cluster config set cloud_topics_enabled=true ---- -Once enabled, create the Cloud Topic: - NOTE: This configuration update requires a restart. [,bash] From 7d5e38e82252e78147b6335da9392760b8307750 Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Tue, 10 Mar 2026 14:58:22 -0500 Subject: [PATCH 63/75] Apply suggestion from @Feediver1 --- modules/develop/pages/manage-topics/cloud-topics.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index 9575a3158f..061c8c9aee 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -68,7 +68,7 @@ IMPORTANT: Any topic specified to be a Cloud Topic cannot be subsequently conver == Enable Cloud Topics for your cluster -To enable Cloud Topics: +To enable Cloud Topics for a cluster: [,bash] ---- From db005d0142c7d10749ed8f1858183ffe029a831c Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Tue, 10 Mar 2026 16:22:21 -0500 Subject: [PATCH 64/75] Apply suggestion from @Feediver1 --- modules/develop/pages/manage-topics/cloud-topics.adoc | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index 061c8c9aee..d027dfb6cd 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -46,16 +46,16 @@ To check your license status, run: ---- rpk cluster license info ---- - -If you intend to use Docker Compose while working with Cloud Topics, see the https://docs.docker.com/compose/install/[Docker Compose installation documentation^]. To verify you have it installed, run: - ++ +- If you intend to use Docker Compose while working with Cloud Topics, see the https://docs.docker.com/compose/install/[Docker Compose installation documentation^]. To verify you have it installed, run: ++ [source,bash] ---- docker compose version ---- - ++ You should see the installed Docker Compose version. - ++ For more details about using Docker Compose with Redpanda, see the xref:get-started:quick-start.adoc[Redpanda Self-Managed Quickstart]. endif::[] From 37b58cc3bbe209d2869b582113c3b546540c792c Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Tue, 10 Mar 2026 16:35:19 -0500 Subject: [PATCH 65/75] Apply suggestion from @Feediver1 --- modules/develop/pages/manage-topics/cloud-topics.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index d027dfb6cd..a78d85443d 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -21,7 +21,7 @@ With Cloud Topics, data from the client is not acknowledged until it is uploaded == Prerequisites -- xref:get-started:rpk-install.adoc[] v25.3 or later. +- xref:get-started:rpk-install.adoc[] v26.1 or later. ifndef::env-cloud[] - xref:manage:tiered-storage.adoc#set-up-tiered-storage[Enable Tiered Storage] on your Redpanda cluster. + From 4b3727b46a4534d830f3932af029c96d41d748a6 Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Tue, 10 Mar 2026 16:42:58 -0500 Subject: [PATCH 66/75] Apply suggestion from Willem --- .../develop/pages/manage-topics/cloud-topics.adoc | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index a78d85443d..f88fd9e17f 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -27,13 +27,15 @@ ifndef::env-cloud[] + [NOTE] ==== -When you enable Tiered Storage at the cluster level, by default, all new topics are Tiered Storage (or standard) topics. However, if you plan to use Cloud Topics for all new topics in a Redpanda cluster, be sure to set the following cluster-level properties to `false`: - -* config_ref:cloud_storage_enable_remote_write,false,properties/object-storage-properties[] -* config_ref:cloud_storage_enable_remote_read,false,properties/object-storage-properties[] - +When you enable Tiered Storage at the cluster level, by default, all new topics are Tiered Storage (or standard) topics. However, if you plan to use Cloud Topics for all new topics in a Redpanda cluster, be sure to set the following cluster-level property: ++ +[,bash] +---- +default_redpanda_storage_mode=cloud +---- ++ This ensures that newly-created Redpanda topics are Cloud Topics by default. - ++ For details, see xref:manage:tiered-storage.adoc#enable-tiered-storage-for-a-cluster[Enable Tiered Storage for a cluster]. ==== From 4c454e02e855fbc18765d03eea39499f2996fe10 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 10 Mar 2026 18:39:46 -0400 Subject: [PATCH 67/75] fix(doc-detective): replace data-testid nav selectors with href-based selectors Console v3.5.2 rewrote the sidebar using TanStack Router with no data-testid attributes on nav items. Nav links now render as standard elements. Update all five Doc Detective find steps to use nav a[href='...'] selectors that match the new sidebar structure. Pulls in fix from #1585. Co-Authored-By: Claude Sonnet 4.6 --- modules/get-started/pages/quick-start.adoc | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/modules/get-started/pages/quick-start.adoc b/modules/get-started/pages/quick-start.adoc index a764ac9c0f..5173dcd955 100644 --- a/modules/get-started/pages/quick-start.adoc +++ b/modules/get-started/pages/quick-start.adoc @@ -206,7 +206,7 @@ Suppose you're asked to find all transactions related to the `.edu` domain. You . In the menu, click *Topics*. // (step {"goTo": "http://localhost:8080/overview"}) -// (step {"find": {"selector": "[data-testid='nav-link-icon-Topics']", "click": true, "timeout": 10000}}) +// (step {"find": {"selector": "nav a[href='/topics']", "click": true, "timeout": 10000}}) . Click the *transactions* topic. When the topic's page opens, the *Messages* tab is selected by default. // (step {"goTo": "http://localhost:8080/topics/transactions"}) . Click *Add filter* > *JavaScript Filter*. @@ -242,7 +242,7 @@ See also: xref:console:ui/programmable-push-filters.adoc[]. On the *Schema Registry* page, you see an overview of your schemas. You can create, manage, and inspect your schemas without leaving Redpanda Console. -// (step {"find": {"selector": "[data-testid='nav-link-icon-Schema Registry']", "click": true, "timeout": 10000}}) +// (step {"find": {"selector": "nav a[href='/schema-registry']", "click": true, "timeout": 10000}}) // (step {"goTo": "http://localhost:8080/schema-registry/subjects/transactions?version=latest"}) // (step {"screenshot": {"path": "../../modules/console/images/schema-reg.png", "maxVariation": 0.10, "overwrite": "aboveVariation"}}) @@ -269,7 +269,7 @@ On the *Security* page, you can: Suppose you're onboarding a new team member who needs access to specific topics. You can use Redpanda Console to ensure users have the right permissions to perform their tasks. . In the menu, click *Security*. -// (step {"find": {"selector": "[data-testid='nav-link-icon-Security']", "click": true, "timeout": 10000}}) +// (step {"find": {"selector": "nav a[href='/security']", "click": true, "timeout": 10000}}) // (step {"goTo": "http://localhost:8080/security/users"}) . On the *Users* tab, click *Create user*. // (step {"find": {"selector": "[data-testid='create-user-button']", "elementText": "Create user", "click": true, "timeout": 100000}}) @@ -383,7 +383,7 @@ Data transforms let you run common data streaming tasks on the Redpanda broker, This quickstart deployment comes with one transform function called `regex` running in your cluster. Its job is to find records in the `logins` topic that contain email addresses with the `.edu` domain and add those to a new topic called `edu-filtered-domains`. In the menu, click *Transforms*. -// (step {"find": { "selector": "[data-testid='nav-link-icon-Transforms']", "click": true, "timeout": 15000}}) +// (step {"find": {"selector": "nav a[href='/transforms']", "click": true, "timeout": 15000}}) On the *Transforms* page, you see your transform. You can use Redpanda Console to manage and monitor your transforms. // (step {"screenshot": {"path": "../../modules/console/images/transforms.png", "maxVariation": 0.10, "overwrite": "aboveVariation"}}) @@ -405,7 +405,7 @@ Audit logs provide a record of all user actions. You can use these logs to track . In the menu, click *Topics*. // (step {"goTo": "http://localhost:8080/overview"}) -// (step {"find": { "selector": "[data-testid='nav-link-icon-Topics']", "click": true, "timeout": 15000}}) +// (step {"find": {"selector": "nav a[href='/topics']", "click": true, "timeout": 15000}}) // (step {"goTo": "http://localhost:8080/topics"}) . Click the *Show internal topics* checkbox. // (step {"find": { "selector": "[data-testid='show-internal-topics-checkbox']", "elementText": "Show internal topics", "click": true, "timeout": 15000}}) From 168f3a95c284b02ca9d795fcb2435f8727234c9a Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Tue, 10 Mar 2026 17:51:53 -0500 Subject: [PATCH 68/75] Apply suggestion from @Feediver1 --- modules/develop/pages/manage-topics/cloud-topics.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index f88fd9e17f..003f837904 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -1,6 +1,6 @@ = Manage Cloud Topics :description: Cloud Topics are Redpanda topics that enable users to trade off latency for lower costs. -:page-beta: true +:page-beta: false :learning-objective-1: Describe the latency and cost trade-offs of Cloud Topics compared to standard Redpanda topics :learning-objective-2: Create a Cloud Topic using rpk after enabling Cloud Topics on your cluster :learning-objective-3: Identify Cloud Topics limitations and configurations that reduce cross-AZ networking costs From 5361a210b70b5f2c813155d60cb2e60016aa5d14 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 10 Mar 2026 19:02:33 -0400 Subject: [PATCH 69/75] chore: retrigger build From 672cab85028c58158ba73c9e47289a6ed4682e2d Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Tue, 10 Mar 2026 18:15:27 -0500 Subject: [PATCH 70/75] Apply suggestion from @Feediver1 --- modules/develop/pages/manage-topics/cloud-topics.adoc | 1 - 1 file changed, 1 deletion(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index 003f837904..4eb9f78fa7 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -1,6 +1,5 @@ = Manage Cloud Topics :description: Cloud Topics are Redpanda topics that enable users to trade off latency for lower costs. -:page-beta: false :learning-objective-1: Describe the latency and cost trade-offs of Cloud Topics compared to standard Redpanda topics :learning-objective-2: Create a Cloud Topic using rpk after enabling Cloud Topics on your cluster :learning-objective-3: Identify Cloud Topics limitations and configurations that reduce cross-AZ networking costs From 744c7b7196c82d49b3e78d6bd4e64a838186f3a3 Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Tue, 10 Mar 2026 18:19:04 -0500 Subject: [PATCH 71/75] Apply suggestion from @Feediver1 --- modules/shared/partials/tristate-behavior-change-25-3.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/shared/partials/tristate-behavior-change-25-3.adoc b/modules/shared/partials/tristate-behavior-change-25-3.adoc index 064f496d63..cce6a25c22 100644 --- a/modules/shared/partials/tristate-behavior-change-25-3.adoc +++ b/modules/shared/partials/tristate-behavior-change-25-3.adoc @@ -1,5 +1,5 @@ Starting in Redpanda v25.3, several topic properties support enhanced tristate behavior. Properties like `retention.ms`, `retention.bytes`, `segment.ms`, and others now distinguish between zero values (immediate eligibility for cleanup/compaction) and negative values (disable the feature entirely). Previously, zero and negative values were treated the same way. ifndef::env-cloud[] -For the complete list of affected properties and detailed information, see xref:get-started:release-notes/redpanda.adoc#behavior-changes[Redpanda v25.3 behavior changes]. +For the complete list of affected properties and detailed information, see xref:get-started:release-notes/redpanda.adoc#behavior-changes[Redpanda v26.1 behavior changes]. endif::[] Review your topic configurations if you currently use zero values for these properties. \ No newline at end of file From cb94db6345151ae919ff27124914c059bab3ee41 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 10 Mar 2026 20:23:25 -0400 Subject: [PATCH 72/75] auto-docs: regenerate topic-properties.adoc for v26.1.1-rc2 Adds redpanda.storage.mode (enum: local, tiered, cloud, unset) and other new topic properties from v26.1.1-rc2. Also updates topic-property-mappings.adoc with the new default_redpanda_storage_mode cluster property mapping. Co-Authored-By: Claude Sonnet 4.6 --- .../partials/properties/topic-properties.adoc | 65 +++++++++++++++---- .../properties/topic-property-mappings.adoc | 3 + 2 files changed, 56 insertions(+), 12 deletions(-) diff --git a/modules/reference/partials/properties/topic-properties.adoc b/modules/reference/partials/properties/topic-properties.adoc index 8887c64c8b..4b1cc175c5 100644 --- a/modules/reference/partials/properties/topic-properties.adoc +++ b/modules/reference/partials/properties/topic-properties.adoc @@ -945,11 +945,10 @@ endif::[] // end::category-retention-compaction[] // tag::category-tiered-storage[] +// tag::exclude-from-docs[] === redpanda.cloud_topic.enabled -Enable this topic as a Cloud Topic. Cloud Topics are optimized for high-throughput, cost-sensitive workloads that can tolerate higher latencies compared to standard Kafka topics. - -IMPORTANT: You can only set this property when creating a topic. After a topic is created as a Cloud Topic, it cannot be converted back to a standard Kafka topic. Before creating Cloud Topics, you must enable the cluster-wide xref:reference:properties/cluster-properties.adoc#cloud_topics_enabled[`cloud_topics_enabled`] property. +Configuration property: redpanda.cloud_topic.enabled [cols="1s,2a"] |=== @@ -977,14 +976,9 @@ ifndef::env-cloud[] | Yes endif::[] -| Related topics -| -* xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics] - -* xref:reference:properties/cluster-properties.adoc#cloud_topics_enabled[`cloud_topics_enabled`] - |=== +// end::exclude-from-docs[] // end::category-tiered-storage[] // tag::category-iceberg-integration[] @@ -1555,6 +1549,53 @@ endif::[] // end::category-tiered-storage[] +// tag::category-other[] +=== redpanda.storage.mode + +No description available. + + +[cols="1s,2a"] +|=== +| Property | Value + +| Type +| `string` (enum) + +| Accepted Values +| +ifndef::env-cloud[] +`local`, `tiered`, `cloud`, `unset` +endif::[] +ifdef::env-cloud[] +`local`, `tiered`, `cloud`, `unset` +endif::[] + + +| Corresponding cluster property +| xref:reference:cluster-properties.adoc#default_redpanda_storage_mode[default_redpanda_storage_mode] + +| Default +| +ifdef::env-cloud[] +Available in the Redpanda Cloud Console +endif::[] +ifndef::env-cloud[] +`unset` +endif::[] + +| Nullable +| No + +ifndef::env-cloud[] +| Restored on xref:manage:whole-cluster-restore.adoc[Whole Cluster Restore] +| Yes +endif::[] + +|=== + +// end::category-other[] + // tag::category-schema-registry[] === redpanda.value.schema.id.validation @@ -1715,9 +1756,9 @@ endif::[] * xref:reference:properties/cluster-properties.adoc#default_topic_replication[`default_topic_replication`] -* xref:develop:manage-topics/config-topics.adoc#choose-the-replication-factor[Choose the replication factor] +* xref:develop:config-topics.adoc#choose-the-replication-factor[Choose the replication factor] -* xref:develop:manage-topics/config-topics.adoc#change-the-replication-factor[Change the replication factor] +* xref:develop:config-topics.adoc#change-the-replication-factor[Change the replication factor] * xref:reference:properties/cluster-properties.adoc#default_topic_replication[default_topic_replication] @@ -2111,7 +2152,7 @@ endif::[] | Related topics | -* xref:develop:manage-topics/config-topics.adoc#configure-write-caching[Write caching] +* xref:develop:config-topics.adoc#configure-write-caching[Write caching] * xref:manage:tiered-storage.adoc[Tiered Storage] diff --git a/modules/reference/partials/properties/topic-property-mappings.adoc b/modules/reference/partials/properties/topic-property-mappings.adoc index 238d37a826..6843535cc9 100644 --- a/modules/reference/partials/properties/topic-property-mappings.adoc +++ b/modules/reference/partials/properties/topic-property-mappings.adoc @@ -63,6 +63,9 @@ | <> | xref:./object-storage-properties.adoc#cloud_storage_enable_remote_write[`cloud_storage_enable_remote_write`] +| <> +| xref:./cluster-properties.adoc#default_redpanda_storage_mode[`default_redpanda_storage_mode`] + | <> | xref:./cluster-properties.adoc#retention_bytes[`retention_bytes`] From cb026723867e15069bdefa7a0b31209468f7f1d6 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 10 Mar 2026 20:30:45 -0400 Subject: [PATCH 73/75] docs: add description for redpanda.storage.mode topic property Adds a description for the new redpanda.storage.mode topic property introduced in v26.1.1-rc2 via property-overrides.json. Also fixes a missing comma JSON syntax error in the existing cloud_topics_reconciliation_interval entry. Regenerates topic-properties.adoc and topic-property-mappings.adoc to reflect the new description. Co-Authored-By: Claude Sonnet 4.6 --- docs-data/property-overrides.json | 90 ++++++++++++++++++- .../partials/properties/topic-properties.adoc | 31 +++++-- 2 files changed, 113 insertions(+), 8 deletions(-) diff --git a/docs-data/property-overrides.json b/docs-data/property-overrides.json index 2831b0270e..98c10587de 100644 --- a/docs-data/property-overrides.json +++ b/docs-data/property-overrides.json @@ -468,6 +468,21 @@ "description": "Configure the addressing style that controls how Redpanda formats bucket URLs for S3-compatible object storage.\n\nLeave this property unset (`null`) to use automatic configuration:\n\n* For AWS S3: Redpanda attempts `virtual_host` addressing first, then falls back to `path` style if needed\n* For MinIO: Redpanda automatically uses `path` style regardless of `MINIO_DOMAIN` configuration\n\nSet this property explicitly to override automatic configuration, ensure consistent behavior across deployments, or when using S3-compatible storage that requires a specific URL format.\n\nCAUTION: AWS requires virtual-hosted addressing for buckets created after September 30, 2020. If you use AWS S3 with buckets created after this date, use `virtual_host` addressing.\n\nNOTE: For MinIO deployments, Redpanda defaults to `path` style when this property is unset. To use `virtual_host` addressing with a configured `MINIO_DOMAIN`, set this property explicitly to `virtual_host`. For other S3-compatible storage backends, consult your provider's documentation to determine the required URL style.", "config_scope": "cluster" }, + "cloud_topics_compaction_interval_ms": { + "version": "v26.1.1-rc2" + }, + "cloud_topics_compaction_key_map_memory": { + "version": "v26.1.1-rc2" + }, + "cloud_topics_compaction_max_object_size": { + "version": "v26.1.1-rc2" + }, + "cloud_topics_disable_level_zero_gc_for_tests": { + "version": "v26.1.1-rc2" + }, + "cloud_topics_disable_metastore_flush_loop_for_tests": { + "version": "v26.1.1-rc2" + }, "cloud_topics_disable_reconciliation_loop": { "config_scope": "cluster" }, @@ -487,9 +502,24 @@ "description": "The duration, in milliseconds, for which a cluster-wide epoch is cached locally on each broker.\n\nCaching the epoch locally reduces the need for frequent coordination with the controller. This property controls how long each broker can use a cached epoch value before fetching the latest value.\n\nIncrease this value to reduce coordination overhead in clusters with stable workloads. Decrease it if you need brokers to react more quickly to epoch changes in Tiered Storage.", "version": "v25.3.3" }, + "cloud_topics_fetch_debounce_enabled": { + "version": "v26.1.1-rc2" + }, + "cloud_topics_l1_indexing_interval": { + "version": "v26.1.1-rc2" + }, + "cloud_topics_long_term_flush_interval": { + "version": "v26.1.1-rc2" + }, "cloud_topics_long_term_garbage_collection_interval": { "config_scope": "cluster" }, + "cloud_topics_parallel_fetch_enabled": { + "version": "v26.1.1-rc2" + }, + "cloud_topics_preregistered_object_ttl": { + "version": "v26.1.1-rc2" + }, "cloud_topics_produce_batching_size_threshold": { "config_scope": "cluster" }, @@ -506,13 +536,34 @@ "config_scope": "cluster" }, "cloud_topics_reconciliation_interval": { - "description": "Time interval at which Redpanda reconciles data between short-term local storage and long-term object storage for Cloud Topics. During this reconciliation process, Redpanda optimizes the storage layout of data in short-term storage to improve the cost and performance associated with accessing data. After the reconciliation process has moved data into long-term storage, the data in short-term storage is subject to removal by a garbage collection process." + "description": "Time interval at which Redpanda reconciles data between short-term local storage and long-term object storage for Cloud Topics. During this reconciliation process, Redpanda optimizes the storage layout of data in short-term storage to improve the cost and performance associated with accessing data. After the reconciliation process has moved data into long-term storage, the data in short-term storage is subject to removal by a garbage collection process.", "related_topics": [ "xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics]", "xref:reference:properties/cluster-properties.adoc#cloud_topics_long_term_garbage_collection_interval[`cloud_topics_long_term_garbage_collection_interval`]" ], "config_scope": "cluster" }, + "cloud_topics_reconciliation_max_interval": { + "version": "v26.1.1-rc2" + }, + "cloud_topics_reconciliation_max_object_size": { + "version": "v26.1.1-rc2" + }, + "cloud_topics_reconciliation_min_interval": { + "version": "v26.1.1-rc2" + }, + "cloud_topics_reconciliation_parallelism": { + "version": "v26.1.1-rc2" + }, + "cloud_topics_reconciliation_slowdown_blend": { + "version": "v26.1.1-rc2" + }, + "cloud_topics_reconciliation_speedup_blend": { + "version": "v26.1.1-rc2" + }, + "cloud_topics_reconciliation_target_fill_ratio": { + "version": "v26.1.1-rc2" + }, "cloud_topics_short_term_gc_backoff_interval": { "description": "The interval, in milliseconds, between invocations of the L0 garbage collection work loop when no progress is being made or errors are occurring.\n\nL0 (level-zero) objects are short-term data objects in Tiered Storage that are periodically garbage collected. When GC encounters errors or cannot make progress (for example, if there are no objects eligible for deletion), this backoff interval prevents excessive retries.\n\nIncrease this value to reduce system load when GC cannot make progress. Decrease it if you need faster retry attempts after transient errors.", "version": "v25.3.3" @@ -525,6 +576,9 @@ "description": "The minimum age, in milliseconds, of an L0 (level-zero) object before it becomes eligible for garbage collection.\n\nThis grace period delays deletion of L0 objects even after they become eligible based on epoch. The delay provides a safety buffer that can support recovery in cases involving accidental deletion or other operational issues.\n\nIncrease this value to extend the retention window for L0 objects, providing more time for recovery from operational errors. Decrease it to free up object storage space more quickly, but with less protection against accidental deletion.", "version": "v25.3.3" }, + "cloud_topics_upload_part_size": { + "version": "v26.1.1-rc2" + }, "cluster_id": { "description": "Cluster identifier.", "config_scope": "cluster" @@ -698,6 +752,9 @@ ], "config_scope": "cluster" }, + "default_redpanda_storage_mode": { + "version": "v26.1.1-rc2" + }, "delete.retention.ms": { "description": "The retention time for tombstone records in a compacted topic. Redpanda removes tombstone records after the retention limit is exceeded.\n\nIf you have enabled Tiered Storage and set <> or <> for the topic, you cannot enable tombstone removal.\n\nIf both `delete.retention.ms` and the cluster property config_ref:tombstone_retention_ms,true,properties/cluster-properties[] are set, `delete.retention.ms` overrides the cluster level tombstone retention for an individual topic.\n\nThis property supports three states:\n\n* Positive value: Sets the milliseconds to retain tombstone records before removal.\n* 0: Tombstone records are immediately eligible for removal.\n* Negative value: Disables tombstone removal entirely for this topic.", "related_topics": [ @@ -706,6 +763,9 @@ ], "config_scope": "topic" }, + "delete_topic_enable": { + "version": "v26.1.1-rc2" + }, "developer_mode": { "description": "CAUTION: Enabling `developer_mode` isn't recommended for production use.\n\nEnable developer mode, which skips most of the checks performed at startup.", "config_scope": "broker", @@ -1017,6 +1077,9 @@ ], "config_scope": "cluster" }, + "internal_rpc_request_timeout_ms": { + "version": "v26.1.1-rc2" + }, "internal_topic_replication_factor": { "description": "Target replication factor for internal topics.\n\n*Unit*: number of replicas per topic.", "config_scope": "cluster" @@ -1224,6 +1287,9 @@ "description": "Prevents log compaction from removing transaction metadata. Only set this to `true` if you experience stability issues related to transaction cleanup during compaction.", "config_scope": "cluster" }, + "log_compaction_max_priority_wait_ms": { + "version": "v26.1.1-rc2" + }, "log_compaction_merge_max_ranges": { "description": "The maximum range of segments that can be processed in a single round of adjacent segment compaction. If `null` (the default value), no maximum is imposed on the number of ranges that can be processed at once. A value below 1 effectively disables adjacent merge compaction.", "config_scope": "cluster" @@ -1387,6 +1453,9 @@ "config_scope": "broker", "category": "schema-registry" }, + "nested_group_behavior": { + "version": "v26.1.1-rc2" + }, "node_id": { "config_scope": "broker", "category": "redpanda", @@ -1431,6 +1500,9 @@ "description": "The URL pointing to the well-known discovery endpoint for the OIDC provider.", "config_scope": "cluster" }, + "oidc_group_claim_path": { + "version": "v26.1.1-rc2" + }, "oidc_principal_mapping": { "description": "Rule for mapping JWT payload claim to a Redpanda user principal.", "related_topics": [ @@ -1503,6 +1575,9 @@ "description": "Controls when and how Redpanda automatically rebalances partition replicas across brokers. For more information, see xref:manage:cluster-maintenance/cluster-balancing.adoc[partition balancing].\n\nValues:\n\n* `continuous`: Partition balancing happens automatically to maintain optimal performance and availability, based on continuous monitoring for node changes (same as `node_add`) and also high disk usage. This option requires an enterprise license, and it is customized by xref:reference:properties/cluster-properties.adoc#partition_autobalancing_node_availability_timeout_sec[`partition_autobalancing_node_availability_timeout_sec`] and xref:reference:properties/cluster-properties.adoc#partition_autobalancing_max_disk_usage_percent[`partition_autobalancing_max_disk_usage_percent`] properties.\n* `node_add`: Partition balancing happens when a node is added.\n* `off`: Partition balancing is disabled. This option is not recommended for production clusters.", "config_scope": "cluster" }, + "partition_autobalancing_node_autodecommission_timeout_sec": { + "version": "v26.1.1-rc2" + }, "partition_autobalancing_node_availability_timeout_sec": { "related_topics": [ "xref:manage:cluster-maintenance/continuous-data-balancing.adoc[Configure Continuous Data Balancing]" @@ -1689,6 +1764,13 @@ ], "config_scope": "topic" }, + "redpanda.storage.mode": { + "description": "The storage mode for a topic. Determines how topic data is stored and whether it is eligible for upload to object storage.\n\nAccepted values:\n\n* `local`: Topic data is stored only on the broker's local disk. Object storage upload is disabled for the topic, regardless of cluster-level Tiered Storage settings.\n* `tiered`: Topic data is stored on local disk and also uploaded to object storage. Enables xref:manage:tiered-storage.adoc[Tiered Storage] for the topic.\n* `cloud`: Topic data is stored in object storage using the Cloud Topics architecture. Local storage is used only as a write buffer.\n* `unset`: Uses the cluster-level config_ref:default_redpanda_storage_mode,true,properties/cluster-properties[] setting, or falls back to legacy `redpanda.remote.read` and `redpanda.remote.write` topic property behavior for backwards compatibility.\n\nThis property overrides the cluster-wide config_ref:default_redpanda_storage_mode,true,properties/cluster-properties[] setting for individual topics.", + "related_topics": [ + "xref:manage:tiered-storage.adoc[Tiered Storage]" + ], + "config_scope": "topic" + }, "redpanda.value.schema.id.validation": { "description": "Enable validation of the schema ID for values on a record. When enabled, Redpanda validates that the schema ID encoded in the record's value is registered in the Schema Registry according to the configured subject name strategy.", "related_topics": [ @@ -1877,11 +1959,17 @@ "config_scope": "broker", "category": "schema-registry" }, + "schema_registry_avro_use_named_references": { + "version": "v26.1.1-rc2" + }, "schema_registry_enable_authorization": { "description": "Enables ACL-based authorization for Schema Registry requests. When `true`, Schema Registry\nuses ACL-based authorization instead of the default `public/user/superuser` authorization model.", "related_topics": [], "config_scope": "cluster" }, + "schema_registry_enable_qualified_subjects": { + "version": "v26.1.1-rc2" + }, "schema_registry_replication_factor": { "description": "Replication factor for internal `_schemas` topic. If unset, defaults to the xref:../cluster-properties.adoc#default_topic_replication[`default_topic_replication`] cluster property.", "related_topics": [ diff --git a/modules/reference/partials/properties/topic-properties.adoc b/modules/reference/partials/properties/topic-properties.adoc index 4b1cc175c5..5d786d976e 100644 --- a/modules/reference/partials/properties/topic-properties.adoc +++ b/modules/reference/partials/properties/topic-properties.adoc @@ -945,10 +945,11 @@ endif::[] // end::category-retention-compaction[] // tag::category-tiered-storage[] -// tag::exclude-from-docs[] === redpanda.cloud_topic.enabled -Configuration property: redpanda.cloud_topic.enabled +Enable this topic as a Cloud Topic. Cloud Topics are optimized for high-throughput, cost-sensitive workloads that can tolerate higher latencies compared to standard Kafka topics. + +IMPORTANT: You can only set this property when creating a topic. After a topic is created as a Cloud Topic, it cannot be converted back to a standard Kafka topic. Before creating Cloud Topics, you must enable the cluster-wide <> property. [cols="1s,2a"] |=== @@ -976,9 +977,14 @@ ifndef::env-cloud[] | Yes endif::[] +| Related topics +| +* xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics] + +* xref:reference:properties/cluster-properties.adoc#cloud_topics_enabled[`cloud_topics_enabled`] + |=== -// end::exclude-from-docs[] // end::category-tiered-storage[] // tag::category-iceberg-integration[] @@ -1552,8 +1558,16 @@ endif::[] // tag::category-other[] === redpanda.storage.mode -No description available. +The storage mode for a topic. Determines how topic data is stored and whether it is eligible for upload to object storage. + +Accepted values: + +* `local`: Topic data is stored only on the broker's local disk. Object storage upload is disabled for the topic, regardless of cluster-level Tiered Storage settings. +* `tiered`: Topic data is stored on local disk and also uploaded to object storage. Enables xref:manage:tiered-storage.adoc[Tiered Storage] for the topic. +* `cloud`: Topic data is stored in object storage using the Cloud Topics architecture. Local storage is used only as a write buffer. +* `unset`: Uses the cluster-level config_ref:default_redpanda_storage_mode,true,properties/cluster-properties[] setting, or falls back to legacy `redpanda.remote.read` and `redpanda.remote.write` topic property behavior for backwards compatibility. +This property overrides the cluster-wide config_ref:default_redpanda_storage_mode,true,properties/cluster-properties[] setting for individual topics. [cols="1s,2a"] |=== @@ -1592,6 +1606,9 @@ ifndef::env-cloud[] | Yes endif::[] +| Related topics +|xref:manage:tiered-storage.adoc[Tiered Storage] + |=== // end::category-other[] @@ -1756,9 +1773,9 @@ endif::[] * xref:reference:properties/cluster-properties.adoc#default_topic_replication[`default_topic_replication`] -* xref:develop:config-topics.adoc#choose-the-replication-factor[Choose the replication factor] +* xref:develop:manage-topics.adoc#choose-the-replication-factor[Choose the replication factor] -* xref:develop:config-topics.adoc#change-the-replication-factor[Change the replication factor] +* xref:develop:manage-topics.adoc#change-the-replication-factor[Change the replication factor] * xref:reference:properties/cluster-properties.adoc#default_topic_replication[default_topic_replication] @@ -2152,7 +2169,7 @@ endif::[] | Related topics | -* xref:develop:config-topics.adoc#configure-write-caching[Write caching] +* xref:develop:manage-topics.adoc#configure-write-caching[Write caching] * xref:manage:tiered-storage.adoc[Tiered Storage] From dec01e5df84d36a7ba96222ca8f807102cea090b Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 10 Mar 2026 20:42:16 -0400 Subject: [PATCH 74/75] revert: remove redpanda.cloud_topic.enabled update from PR This property is superseded by redpanda.storage.mode and should not be documented here. Reverts to exclude_from_docs: true and removes the cross-reference to it from cloud_topics_enabled. Co-Authored-By: Claude Sonnet 4.6 --- docs-data/property-overrides.json | 11 +++-------- .../partials/properties/topic-properties.adoc | 12 +++--------- 2 files changed, 6 insertions(+), 17 deletions(-) diff --git a/docs-data/property-overrides.json b/docs-data/property-overrides.json index 98c10587de..5ece5d67e0 100644 --- a/docs-data/property-overrides.json +++ b/docs-data/property-overrides.json @@ -489,8 +489,7 @@ "cloud_topics_enabled": { "description": "Enable Cloud Topics for the cluster. Cloud Topics are optimized for high-throughput, cost-sensitive workloads that can tolerate higher latencies compared to standard Kafka topics.", "related_topics": [ - "xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics]", - "xref:reference:properties/topic-properties.adoc#redpandacloudtopicenabled[`redpanda.cloud_topic.enabled`]" + "xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics]" ], "config_scope": "cluster" }, @@ -1657,13 +1656,9 @@ "category": "redpanda" }, "redpanda.cloud_topic.enabled": { - "description": "Enable this topic as a Cloud Topic. Cloud Topics are optimized for high-throughput, cost-sensitive workloads that can tolerate higher latencies compared to standard Kafka topics.\n\nIMPORTANT: You can only set this property when creating a topic. After a topic is created as a Cloud Topic, it cannot be converted back to a standard Kafka topic. Before creating Cloud Topics, you must enable the cluster-wide <> property.", - "related_topics": [ - "xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics]", - "xref:reference:properties/cluster-properties.adoc#cloud_topics_enabled[`cloud_topics_enabled`]" - ], "config_scope": "topic", - "category": "tiered-storage" + "category": "tiered-storage", + "exclude_from_docs": true }, "redpanda.iceberg.delete": { "description": "Whether the corresponding Iceberg table is deleted upon deleting the topic.", diff --git a/modules/reference/partials/properties/topic-properties.adoc b/modules/reference/partials/properties/topic-properties.adoc index 5d786d976e..e89847f356 100644 --- a/modules/reference/partials/properties/topic-properties.adoc +++ b/modules/reference/partials/properties/topic-properties.adoc @@ -945,11 +945,10 @@ endif::[] // end::category-retention-compaction[] // tag::category-tiered-storage[] +// tag::exclude-from-docs[] === redpanda.cloud_topic.enabled -Enable this topic as a Cloud Topic. Cloud Topics are optimized for high-throughput, cost-sensitive workloads that can tolerate higher latencies compared to standard Kafka topics. - -IMPORTANT: You can only set this property when creating a topic. After a topic is created as a Cloud Topic, it cannot be converted back to a standard Kafka topic. Before creating Cloud Topics, you must enable the cluster-wide <> property. +Configuration property: redpanda.cloud_topic.enabled [cols="1s,2a"] |=== @@ -977,14 +976,9 @@ ifndef::env-cloud[] | Yes endif::[] -| Related topics -| -* xref:develop:manage-topics/cloud-topics.adoc[Cloud Topics] - -* xref:reference:properties/cluster-properties.adoc#cloud_topics_enabled[`cloud_topics_enabled`] - |=== +// end::exclude-from-docs[] // end::category-tiered-storage[] // tag::category-iceberg-integration[] From 13dd21adb497494b241e09b6c76de34527939455 Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Wed, 11 Mar 2026 10:58:08 -0500 Subject: [PATCH 75/75] Apply suggestions from code review Co-authored-by: Joyce Fee <102751339+Feediver1@users.noreply.github.com> --- modules/develop/pages/manage-topics/cloud-topics.adoc | 3 --- 1 file changed, 3 deletions(-) diff --git a/modules/develop/pages/manage-topics/cloud-topics.adoc b/modules/develop/pages/manage-topics/cloud-topics.adoc index 4eb9f78fa7..2d0b3b8ed4 100644 --- a/modules/develop/pages/manage-topics/cloud-topics.adoc +++ b/modules/develop/pages/manage-topics/cloud-topics.adoc @@ -27,14 +27,11 @@ ifndef::env-cloud[] [NOTE] ==== When you enable Tiered Storage at the cluster level, by default, all new topics are Tiered Storage (or standard) topics. However, if you plan to use Cloud Topics for all new topics in a Redpanda cluster, be sure to set the following cluster-level property: -+ [,bash] ---- default_redpanda_storage_mode=cloud ---- -+ This ensures that newly-created Redpanda topics are Cloud Topics by default. -+ For details, see xref:manage:tiered-storage.adoc#enable-tiered-storage-for-a-cluster[Enable Tiered Storage for a cluster]. ====