dataplane: Make MachineConfig CRD dependency optional #1757

stuggi · 2026-01-05T13:27:10Z

Problem:
When deploying the openstack-operator on clusters where the OpenShift Machine Config Operator (MCO) is not installed, the openstackdataplanenodeset controller fails to start with the error:

ERROR setup problem running manager {"error": "failed to wait for
openstackdataplanenodeset caches to sync: timed out waiting for cache
to be synced for Kind *v1.MachineConfig"}

This occurs because the controller unconditionally sets up a watch for MachineConfig resources in SetupWithManager(). When the MachineConfig CRD doesn't exist (e.g., on non-OpenShift Kubernetes clusters or clusters without MCO), the informer cache sync times out and the controller fails to start.

Similarly, the disconnected environment detection code (IsDisconnectedOCP) would fail if ImageContentSourcePolicy or ImageDigestMirrorSet CRDs don't exist.

Solution:
Implement conditional/dynamic CRD watching for MachineConfig:

Remove the MachineConfig watch from SetupWithManager() so the controller can start without the CRD being present
Add ensureMachineConfigWatch() function that:
- Checks if the MachineConfig CRD exists by querying apiextensions.k8s.io/v1/CustomResourceDefinition
- Dynamically adds the watch using Controller.Watch() if the CRD exists
- Tracks watched resources to avoid duplicate watch registration
- Logs an informational message if the CRD is not available
Call ensureMachineConfigWatch() at the start of each reconciliation to attempt setting up the watch when the CRD becomes available
Update IsDisconnectedOCP() to handle missing ICSP/IDMS CRDs gracefully instead of returning an error
Update inventory generation to log a warning instead of failing when MachineConfig is not available in a disconnected environment

This allows the operator to:

Start successfully even without the MachineConfig CRD
Work on non-OpenShift Kubernetes clusters
Gracefully degrade disconnected environment support
Automatically enable MachineConfig watching if the CRD becomes available later

The MachineConfig watch is used for disconnected/mirrored environments to detect changes to the 99-master-generated-registries MachineConfig and propagate registry configuration to dataplane nodes. This feature is now optional rather than required.

Manual registry configuration without MCO:
If the MachineConfig CRD is not available but you need to configure container registries on dataplane nodes, you can set the ansible variables directly in the OpenStackDataPlaneNodeSet spec:

  spec:
    nodeTemplate:
      ansible:
        ansibleVars:
          edpm_podman_disconnected_ocp: true
          edpm_podman_registries_conf: |
            unqualified-search-registries = ["registry.access.redhat.com"]
            [[registry]]
              prefix = ""
              location = "quay.io/openstack-k8s-operators"
              [[registry.mirror]]
                location = "my-registry.example.com/openstack-k8s-operators"

Fixes: cache sync timeout when MachineConfig CRD is not installed

Related: OSPRH-24026

dprince · 2026-01-05T21:23:33Z

approach here seems reasonable to me. will let others weigh in

stuggi · 2026-01-07T07:47:45Z

/retest

abays

/lgtm

abays · 2026-01-07T10:24:16Z

Deferring to @slagle for final approval

slagle

lgtm, no problems with the change.
I have an inline comment for potential improvement (assuming I'm reading it correctly).

slagle · 2026-01-07T23:02:50Z

internal/dataplane/inventory.go

+			helper.GetLogger().Info("Disconnected environment detected but MachineConfig not available. "+
+				"Registry configuration will not be propagated to dataplane nodes. "+
+				"You may need to configure registries.conf manually.",
+				"error", err.Error())


This is probably ok for now, but I think we should file an issue to:

if isDisconnected and <<MachineConfig CRD doesn't exist>>; then error and fail reconcile

Because I think it's pretty likely that this logged error goes completely unnoticed. Previously we failed hard if there was an error, now we just log and continue.

Should isNoMatchError be used to check if the error was actually a MachineConfig CRD not found error?

stuggi · 2026-01-08T08:14:23Z

internal/dataplane/inventory.go

+			if util.IsNoMatchError(err) {
+				helper.GetLogger().Info("Disconnected environment detected but MachineConfig CRD not available. "+
+					"Registry configuration will not be propagated to dataplane nodes. "+
+					"You may need to configure registries.conf manually using ansibleVars "+
+					"(edpm_podman_disconnected_ocp and edpm_podman_registries_conf).",
+					"error", err.Error())
+			} else {
+				// CRD exists but resource not found, or other errors (network issues,
+				// permissions, etc.) - return the error. If MCO is installed but the
+				// registry MachineConfig doesn't exist, this indicates a misconfiguration.
+				return "", fmt.Errorf("failed to get MachineConfig registry configuration: %w", err)


@slagle iiuc this is what you meant, right?

yea, looks good

Problem: When deploying the openstack-operator on clusters where the OpenShift Machine Config Operator (MCO) is not installed, the openstackdataplanenodeset controller fails to start with the error: ERROR setup problem running manager {"error": "failed to wait for openstackdataplanenodeset caches to sync: timed out waiting for cache to be synced for Kind *v1.MachineConfig"} This occurs because the controller unconditionally sets up a watch for MachineConfig resources in SetupWithManager(). When the MachineConfig CRD doesn't exist (e.g., on non-OpenShift Kubernetes clusters or clusters without MCO), the informer cache sync times out and the controller fails to start. Similarly, the disconnected environment detection code (IsDisconnectedOCP) would fail if ImageContentSourcePolicy or ImageDigestMirrorSet CRDs don't exist. Solution: Implement conditional/dynamic CRD watching for MachineConfig: 1. Remove the MachineConfig watch from SetupWithManager() so the controller can start without the CRD being present 2. Add ensureMachineConfigWatch() function that: - Checks if the MachineConfig CRD exists by querying apiextensions.k8s.io/v1/CustomResourceDefinition - Dynamically adds the watch using Controller.Watch() if the CRD exists - Tracks watched resources to avoid duplicate watch registration - Logs an informational message if the CRD is not available 3. Call ensureMachineConfigWatch() at the start of each reconciliation to attempt setting up the watch when the CRD becomes available 4. Update IsDisconnectedOCP() to handle missing ICSP/IDMS CRDs gracefully instead of returning an error 5. Update inventory generation error handling to distinguish between: - IsNoMatchError (CRD not installed): Log warning and continue. This is expected on non-OpenShift clusters or clusters without MCO. - IsNotFound (CRD exists but resource missing): Return error. If MCO is installed and a disconnected environment is detected, the registry MachineConfig should exist. Missing resource indicates misconfiguration. This allows the operator to: - Start successfully even without the MachineConfig CRD - Work on non-OpenShift Kubernetes clusters - Gracefully degrade disconnected environment support - Automatically enable MachineConfig watching if the CRD becomes available later - Report actual misconfigurations when MCO is present but registry MachineConfig is missing The MachineConfig watch is used for disconnected/mirrored environments to detect changes to the 99-master-generated-registries MachineConfig and propagate registry configuration to dataplane nodes. This feature is now optional rather than required. Manual registry configuration without MCO: If the MachineConfig CRD is not available but you need to configure container registries on dataplane nodes, you can set the ansible variables directly in the OpenStackDataPlaneNodeSet spec: spec: nodeTemplate: ansible: ansibleVars: edpm_podman_disconnected_ocp: true edpm_podman_registries_conf: | unqualified-search-registries = ["registry.access.redhat.com"] [[registry]] prefix = "" location = "quay.io/openstack-k8s-operators" [[registry.mirror]] location = "my-registry.example.com/openstack-k8s-operators" Fixes: cache sync timeout when MachineConfig CRD is not installed Related: OSPRH-24026 Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Martin Schuppert <mschuppert@redhat.com>

Add comprehensive unit tests for the disconnected environment detection and registry configuration retrieval functions in the util package. Tests cover: - IsNoMatchError: Detects both 'no matches for kind' (real API server) and 'no kind is registered' (fake client) errors - IsDisconnectedOCP: Tests with ICSP, IDMS, both, empty lists, and missing CRDs (graceful degradation) - GetMCRegistryConf: Tests successful retrieval, missing MachineConfig, missing CRD, invalid ignition format, missing base64 prefix, and invalid base64 content - Full scenario tests for disconnected environments and non-OpenShift cluster graceful degradation The IsNoMatchError function is updated to also detect 'no kind is registered' errors which occur in functional tests where the fake client behaves differently than a real Kubernetes API server. The MachineConfigNotFound test verifies that IsNotFound errors are NOT treated as IsNoMatchError - when the CRD exists but the resource doesn't, this should be an error (not a warning), indicating misconfiguration.

softwarefactory-project-zuul · 2026-01-08T08:24:29Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/42ded77318af4a1db77f49124441f538

❌ openstack-k8s-operators-content-provider FAILURE in 10m 00s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ openstack-operator-tempest-multinode SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider

stuggi · 2026-01-08T09:42:25Z

/retest

slagle · 2026-01-08T13:13:01Z

internal/dataplane/inventory.go

+			if util.IsNoMatchError(err) {
+				helper.GetLogger().Info("Disconnected environment detected but MachineConfig CRD not available. "+
+					"Registry configuration will not be propagated to dataplane nodes. "+
+					"You may need to configure registries.conf manually using ansibleVars "+
+					"(edpm_podman_disconnected_ocp and edpm_podman_registries_conf).",
+					"error", err.Error())
+			} else {
+				// CRD exists but resource not found, or other errors (network issues,
+				// permissions, etc.) - return the error. If MCO is installed but the
+				// registry MachineConfig doesn't exist, this indicates a misconfiguration.
+				return "", fmt.Errorf("failed to get MachineConfig registry configuration: %w", err)


yea, looks good

openshift-ci · 2026-01-08T13:13:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: slagle, stuggi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [slagle,stuggi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

stuggi added do-not-merge/work-in-progress do-not-merge/hold labels Jan 5, 2026

openshift-ci bot removed the do-not-merge/work-in-progress label Jan 5, 2026

openshift-ci bot requested review from olliewalsh and viroel January 5, 2026 13:27

openshift-ci bot added the approved label Jan 5, 2026

stuggi marked this pull request as draft January 5, 2026 13:27

openshift-ci bot added the do-not-merge/work-in-progress label Jan 5, 2026

stuggi marked this pull request as ready for review January 5, 2026 14:19

openshift-ci bot removed the do-not-merge/work-in-progress label Jan 5, 2026

openshift-ci bot requested review from rebtoor and slagle January 5, 2026 14:19

stuggi removed request for olliewalsh, rebtoor, slagle and viroel January 5, 2026 14:25

stuggi removed the do-not-merge/hold label Jan 7, 2026

stuggi requested a review from slagle January 7, 2026 07:54

abays reviewed Jan 7, 2026

View reviewed changes

openshift-ci bot assigned abays Jan 7, 2026

openshift-ci bot added the lgtm label Jan 7, 2026

abays added the do-not-merge/hold label Jan 7, 2026

slagle reviewed Jan 7, 2026

View reviewed changes

stuggi force-pushed the conditional_crd branch from 1ece03f to 3aef989 Compare January 8, 2026 07:48

openshift-ci bot removed the lgtm label Jan 8, 2026

stuggi force-pushed the conditional_crd branch from 3aef989 to 0d4b629 Compare January 8, 2026 08:13

stuggi commented Jan 8, 2026

View reviewed changes

stuggi and others added 2 commits January 8, 2026 09:21

stuggi force-pushed the conditional_crd branch from 0d4b629 to 0d0d158 Compare January 8, 2026 08:24

slagle approved these changes Jan 8, 2026

View reviewed changes

openshift-ci bot assigned slagle Jan 8, 2026

openshift-ci bot added the lgtm label Jan 8, 2026

stuggi removed the do-not-merge/hold label Jan 8, 2026

openshift-merge-bot bot merged commit c92ad83 into openstack-k8s-operators:main Jan 8, 2026
8 checks passed

dataplane: Make MachineConfig CRD dependency optional #1757

dataplane: Make MachineConfig CRD dependency optional #1757

Conversation

stuggi commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dprince commented Jan 5, 2026

Uh oh!

stuggi commented Jan 7, 2026

Uh oh!

abays left a comment

Choose a reason for hiding this comment

Uh oh!

abays commented Jan 7, 2026

Uh oh!

slagle left a comment

Choose a reason for hiding this comment

Uh oh!

slagle Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

stuggi Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

slagle Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

softwarefactory-project-zuul bot commented Jan 8, 2026

Uh oh!

stuggi commented Jan 8, 2026

Uh oh!

slagle Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Jan 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

stuggi commented Jan 5, 2026 •

edited

Loading