Phasing out Datacenters in favor of Locations

[hakman]

TL;DR

Based on the 2025-12-16 changelog entry, the datacenter property in the request body and response of Servers and Primary IPs is deprecated and the removal will happen after 1 July 2026.
The deprecation already happened in hcloud-go#793.
HCloud CCM should also be updated to reflect this change.

Expected behavior

The current mapping is for K8s topology keys is:

• topology.kubernetes.io/region: Server.Datacenter.Location.Name -> Server.Location.Name
• topology.kubernetes.io/zone: Server.Datacenter.Name - no longer exists.

This will be a breaking change, with the DC name not being exposed anymore.

Not quite sure what the best approach is for topology.kubernetes.io/zone:

• use Server.Location.Name
• drop the topology.kubernetes.io/zone label (I think some providers do that, but may generate scheduling issues).

[lukasmetzner]

Hey @hakman,

thank you for raising this issue. We already discussing this internally right now, what the best approach is. We have both your options in discussion.

Best Regards
Lukas

[hakman]

Thanks for the quick reply @lukasmetzner.

[apricote]

Context

We plan to remove the notion of a “datacenter” from the Primary IP and Cloud Server API resources after 01 July 2026. You can get more information in the changelog entry: Phasing out Datacenters in favor of Locations for Primary IPs and Servers

In HCCM we use the datacenter name for the topology.kubernetes.io/zone label on Kubernetes nodes. We also rely on this “mapping” in other Kubernetes integrations. Additionally, customers may use this label in their “Pod Topology Spread Constraints”, so we can not easily drop it. The Kubernetes node label is not updated by HCCM, after the node got first initialized.

Well-known labels

Kubernetes defines the topology labels topology.kubernetes.io/zone and topology.kubernetes.io/region as follows (source):

A zone represents a logical failure domain. It is common for Kubernetes clusters to span multiple zones for increased availability. While the exact definition of a zone is left to infrastructure implementations, common properties of a zone include very low network latency within a zone, no-cost network traffic within a zone, and failure independence from other zones. For example, nodes within a zone might share a network switch, but nodes in different zones should not.

A region represents a larger domain, made up of one or more zones. It is uncommon for Kubernetes clusters to span multiple regions, While the exact definition of a zone or region is left to infrastructure implementations, common properties of a region include higher network latency between them than within them, non-zero cost for network traffic between them, and failure independence from other zones or regions. For example, nodes within a region might share power infrastructure (e.g. a UPS or generator), but nodes in different regions typically would not.

Additionally, it places this constraint on the labels:

It should be safe to assume that topology labels do not change. Even though labels are strictly mutable, consumers of them can assume that a given node is not going to be moved between zones without being destroyed and recreated.

Where do we use the datacenter name?

• hcloud-cloud-controller-manager

  • hcloud-cloud-controller-manager/hcloud/instances.go

    Line 351 in b726bef

    Zone: s.Datacenter.Name,

  • Sets the label topology.kubernetes.io/zone on new nodes (through the k8s.io/cloud-provider library)
  • Additionally, for Robot servers we also set the datacenter name here, which will not go away.

	Current
Cloud Server	topology.kubernetes.io/zone: fsn1-dc14 topology.kubernetes.io/region: fsn1
Robot Server	topology.kubernetes.io/zone: fsn1-dc7 topology.kubernetes.io/region: fsn1

• cluster-autoscaler Provider
  • The scheduler needs to know the labels of potential nodes, to make sure that the pending pods can actually be scheduled onto it. For this, the Hetzner provider “calculates” the expected labels of the node. Luckily we only set the region label here with the location name.

	Current
Cloud Server	topology.kubernetes.io/region: fsn1
Robot Server	Not considered in cluster-autoscaler

Options

These options primarily look at the changes in hcloud-cloud-controller-manager. The implementation in the cluster-autoscaler Provider may need to be adjusted as well, but is not explicitly mentioned.

Option 1: Keep historical Location→DC mapping in our code

We can store a static map from location → DC, as this information is static and does not change anymore. There is a single DC per location. If there is a new location, we will use the location name for both the zone and region.

	Current	Suggestion
Cloud Server	topology.kubernetes.io/zone: fsn1-dc14 topology.kubernetes.io/region: fsn1	Existing Location: topology.kubernetes.io/zone: fsn1-dc14 topology.kubernetes.io/region: fsn1 New Location: topology.kubernetes.io/zone: ffm topology.kubernetes.io/region: ffm
Robot Server	topology.kubernetes.io/zone: fsn1-dc7 topology.kubernetes.io/region: fsn1	topology.kubernetes.io/zone: fsn1-dc7 topology.kubernetes.io/region: fsn1

Pros:

• Pretty simple to do and not breaking for the user
• Even for new locations there is still a 1:1 mapping between region:zone labels, same as before with the DC names

Cons:

• We keep historical baggage

Option 1.1: Do not set zone label on cloud nodes

Add a disabled-by-default option to not set zone label any more and only keep the region, which contains the location information (breaking change).

	Current	Flag Disabled (default)	Flag Enabled
Cloud Server	topology.kubernetes.io/zone: fsn1-dc14 topology.kubernetes.io/region: fsn1	topology.kubernetes.io/zone: fsn1-dc14 topology.kubernetes.io/region: fsn1	topology.kubernetes.io/region: fsn1
Robot Server	topology.kubernetes.io/zone: fsn1-dc7 topology.kubernetes.io/region: fsn1	topology.kubernetes.io/zone: fsn1-dc7 topology.kubernetes.io/region: fsn1	topology.kubernetes.io/zone: fsn1-dc7 topology.kubernetes.io/region: fsn1

Pros:

• New installations can disable the historical baggage
• Users can set this option on new clusters in preparation for a potential next HCCM major version, where the default behavior is most likely changed

Cons:

• It still makes sense to set the Zone label for Robot nodes, so we have some nodes with the label and some nodes without
• The default podTopologySpreadConstraints would cause the scheduler to prefer nodes with the topology.kubernetes.io/zone label, so Robot nodes would get filled to exhaustion before Cloud nodes are being considered for scheduling

Option 1.2: Change mapping from DC+Location to Location+Network Zone

Add feature flag to change the move the mapping “one level higher”, drop the datacenter and add the network zone.

• Zone: datacenter → location
• Region: location → network-zone

	Current	Flag Disabled (default)	Flag Enabled
Cloud Server	topology.kubernetes.io/zone: fsn1-dc14 topology.kubernetes.io/region: fsn1	topology.kubernetes.io/zone: fsn1-dc14 topology.kubernetes.io/region: fsn1	topology.kubernetes.io/zone: fsn1 topology.kubernetes.io/region: eu-central
Robot Server	topology.kubernetes.io/zone: fsn1-dc7 topology.kubernetes.io/region: fsn1	topology.kubernetes.io/zone: fsn1-dc7 topology.kubernetes.io/region: fsn1	topology.kubernetes.io/zone: fsn1-dc7 topology.kubernetes.io/region: eu-central

Pros:

• Users can set this option on new clusters in preparation for a potential next HCCM major version, where the default behavior is most likely changed

Cons:

• Feature flag needs to be added to HCCM and cluster-autoscaler and users need to set both or none, otherwise autoscaling may break
• We have to use an inconsistent schema between Robot and Cloud, otherwise we are loosing information
  • Robot has actual DC information, which the user might want to use → This must be part of the zone label then
  • ⚠️ Because of this, it would not be possible to say “distribute pods between the locations” because robot servers from different DCs would count as “different locations”.

Option 2: Announce breaking change and put location name in zone label

We could accept the breaking change from the API and forward it to the customers by removing datacenter for cloud servers from our code, and passing along the location name in zone. The users do not really lose any information, because of the 1:1 mapping between (virtual) DCs and locations. If the user was relying on the zone label for anything, they would need to act and update their usage of the field.

We could provide documentation and scripts to:

• Check the kubernetes cluster for any uses of the zone label and warn the customer
  • Pod Topology Spread Constraints, Node Selector, Node Affinity
• Migrate existing nodes to the new label value, if desired

	Current	Suggestions
Cloud Server	topology.kubernetes.io/zone: fsn1-dc14 topology.kubernetes.io/region: fsn1	Old Node: topology.kubernetes.io/zone: fsn1-dc14 topology.kubernetes.io/region: fsn1 New Node: topology.kubernetes.io/zone: fsn1 topology.kubernetes.io/region: fsn1
Robot Server	topology.kubernetes.io/zone: fsn1-dc7 topology.kubernetes.io/region: fsn1	topology.kubernetes.io/zone: fsn1-dc7 topology.kubernetes.io/region: fsn1

Pros:

• We fill zone and region with the most sensible values matching their explanations from Kubernetes docs, while staying consistent with Robot servers. We keep the same information available.

Cons:

• By using the zone label for Cloud Servers we would have a harder time “re-introducing” datacenters in the future if it becomes necessary.
• Hard breaking change for customers
  • Partially mitigated by docs and scripts to help the user, though we can never be sure if we identified all uses of the label

Docker Image

We currently have a latest tag of HCCM, which is not actually latest , but points to a snapshot release of v1.14. This version will break after July 2026, so we could delete the latest tag from the repo.

Decision

We will implement Option 1 immediately, and add a static mapping to the code (#1159). This is chosen as it is the quickest to implement and has the same behaviour as previous versions.

We will take a look at Option 1.1 and Option 2 in the coming weeks, and possibly implement them as disabled-by-default flags, with the intent to change the default in a possible major version.

[hakman]

Thanks for the update @apricote.
Looking forward to see the decision on 1.1 and 2.