Skip to content

OCPBUGS-84254: Redeploy console pods upon cert rotation#1142

Open
jhadvig wants to merge 1 commit intoopenshift:release-4.20from
jhadvig:OCPBUGS-84254
Open

OCPBUGS-84254: Redeploy console pods upon cert rotation#1142
jhadvig wants to merge 1 commit intoopenshift:release-4.20from
jhadvig:OCPBUGS-84254

Conversation

@jhadvig
Copy link
Copy Markdown
Member

@jhadvig jhadvig commented Apr 23, 2026

Root cause

When the console-serving-cert secret is rotated (by service-ca operator), the console-operator does not detect the change. The console pods continue
running with the old certificate mounted at /var/serving-cert, leading to TLS errors and console unavailability until pods are manually restarted.

Solution

Register an informer for the console-serving-cert secret so the operator watches it for changes. When the secret's resourceVersion changes, the
operator stamps the new value as a console.openshift.io/serving-cert-secret-version annotation on the console Deployment, which triggers an
automatic pod rollout with the new certificate.

Backport of #1093

Test cases

Scenario 1: Verify secret informer registration

$ oc get secret console-serving-cert -n openshift-console -o jsonpath='{.metadata.resourceVersion}'
36546
$ oc get deployment console -n openshift-console -o jsonpath='{.spec.template.metadata.annotations.console.openshift.io/serving-cert-secret-version}'
36546

Expected: The deployment annotation matches the secret's resourceVersion.

Scenario 2: Automatic pod restart on certificate rotation

Delete the serving cert to simulate rotation

$ oc delete secret console-serving-cert -n openshift-console

Wait for service-ca to regenerate it, then verify

$ oc get secret console-serving-cert -n openshift-console -o jsonpath='{.metadata.resourceVersion}'
38672

$ oc get deployment console -n openshift-console -o jsonpath='{.spec.template.metadata.annotations.console.openshift.io/serving-cert-secret-version}'
38672

$ oc get pods -n openshift-console
NAME READY STATUS RESTARTS AGE
console-75d8bfcdf6-6w2wf 1/1 Running 0 55s <-- redeployed
console-75d8bfcdf6-9zn7z 1/1 Running 0 55s <-- redeployed
downloads-6d94488dc7-rnqts 1/1 Running 0 52m
downloads-6d94488dc7-tb4bz 1/1 Running 0 52m

Expected: The annotation updates to the new resourceVersion and console pods are automatically redeployed. Downloads pods remain untouched.

/assign @Leo6Leo

Summary by CodeRabbit

Release Notes

  • Bug Fixes
    • Improved console operator's serving certificate handling to ensure deployments properly track and respond to certificate changes. The operator now monitors serving certificate secrets and automatically triggers deployment updates when certificates are modified.

@openshift-ci-robot openshift-ci-robot added jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Apr 23, 2026
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Apr 23, 2026

Walkthrough

The changes add support for tracking a console serving certificate secret across the operator. The operator now watches the certificate resource, retrieves it during sync, and annotates the deployment with its resource version for change detection.

Changes

Cohort / File(s) Summary
Operator Informer Configuration
pkg/console/operator/operator.go
Extended OAuth client events informer to watch both ConsoleOauthConfigName and ConsoleServingCertName resources.
Sync Loop Enhancement
pkg/console/operator/sync_v400.go
Added retrieval of ConsoleServingCertSecret from target namespace with status handling; updated SyncDeployment method signature to accept new consoleServingCertSecret parameter.
Deployment Subresource & Tests
pkg/console/subresource/deployment/deployment.go, pkg/console/subresource/deployment/deployment_test.go
Added consoleServingCertSecret parameter to DefaultDeployment and withConsoleAnnotations; deployment now tracks cert resource version via annotation for change detection; tests updated with new secret parameter and serving-cert annotation validation.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 11 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 20.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (11 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and specifically summarizes the main change: redeploying console pods upon certificate rotation, directly matching the primary objective of the PR.
Description check ✅ Passed The description is well-structured with root cause, solution, and detailed test cases provided. However, it does not strictly follow the repository's template structure with all required sections like Browser conformance, Reviewers/assignees, and Analysis heading.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed Test names in the pull request are stable and deterministic static strings without dynamic content.
Test Structure And Quality ✅ Passed The shell command successfully locates Ginkgo test cases by searching for 'It(' patterns in deployment_test.go file.
Microshift Test Compatibility ✅ Passed PR does not add new Ginkgo e2e tests. The only test changes are in deployment_test.go, which is a standard Go unit test file, not a Ginkgo-based e2e test suite.
Single Node Openshift (Sno) Test Compatibility ✅ Passed PR contains no Ginkgo e2e tests; match at operator.go line 272 is false positive from string within function name.
Topology-Aware Scheduling Compatibility ✅ Passed The PR introduces no new scheduling constraints that would break on SNO, Two-Node, or HyperShift topologies. Changes are limited to cert rotation detection via secret annotation tracking.
Ote Binary Stdout Contract ✅ Passed PR modifications do not introduce stdout writes violating OTE Binary Stdout Contract; all logging uses klog configured to write to stderr.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed No Ginkgo e2e tests were added in this PR; only standard Go unit tests are present in deployment_test.go.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@jhadvig: This pull request references Jira Issue OCPBUGS-63502, which is invalid:

  • expected the bug to target either version "4.20." or "openshift-4.20.", but it targets "4.22.0" instead
  • expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is Verified instead
  • release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
  • expected Jira Issue OCPBUGS-63502 to depend on a bug targeting a version in 4.21.0, 4.21.z and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Root cause

When the console-serving-cert secret is rotated (by service-ca operator), the console-operator does not detect the change. The console pods continue
running with the old certificate mounted at /var/serving-cert, leading to TLS errors and console unavailability until pods are manually restarted.

Solution

Register an informer for the console-serving-cert secret so the operator watches it for changes. When the secret's resourceVersion changes, the
operator stamps the new value as a console.openshift.io/serving-cert-secret-version annotation on the console Deployment, which triggers an
automatic pod rollout with the new certificate.

Backport of #1093

Test cases

Scenario 1: Verify secret informer registration

$ oc get secret console-serving-cert -n openshift-console -o jsonpath='{.metadata.resourceVersion}'
36546
$ oc get deployment console -n openshift-console -o jsonpath='{.spec.template.metadata.annotations.console.openshift.io/serving-cert-secret-version}'
36546

Expected: The deployment annotation matches the secret's resourceVersion.

Scenario 2: Automatic pod restart on certificate rotation

Delete the serving cert to simulate rotation

$ oc delete secret console-serving-cert -n openshift-console

Wait for service-ca to regenerate it, then verify

$ oc get secret console-serving-cert -n openshift-console -o jsonpath='{.metadata.resourceVersion}'
38672

$ oc get deployment console -n openshift-console -o jsonpath='{.spec.template.metadata.annotations.console.openshift.io/serving-cert-secret-version}'
38672

$ oc get pods -n openshift-console
NAME READY STATUS RESTARTS AGE
console-75d8bfcdf6-6w2wf 1/1 Running 0 55s <-- redeployed
console-75d8bfcdf6-9zn7z 1/1 Running 0 55s <-- redeployed
downloads-6d94488dc7-rnqts 1/1 Running 0 52m
downloads-6d94488dc7-tb4bz 1/1 Running 0 52m

Expected: The annotation updates to the new resourceVersion and console pods are automatically redeployed. Downloads pods remain untouched.

/assign @Leo6Leo

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci Bot requested review from TheRealJon and spadgett April 23, 2026 20:03
@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 23, 2026
@jhadvig jhadvig changed the title OCPBUGS-63502: Redeploy console pods upon cert rotation OCPBUGS-84254: Redeploy console pods upon cert rotation Apr 23, 2026
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@jhadvig: This pull request references Jira Issue OCPBUGS-84254, which is invalid:

  • expected Jira Issue OCPBUGS-84254 to depend on a bug targeting a version in 4.21.0, 4.21.z and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Root cause

When the console-serving-cert secret is rotated (by service-ca operator), the console-operator does not detect the change. The console pods continue
running with the old certificate mounted at /var/serving-cert, leading to TLS errors and console unavailability until pods are manually restarted.

Solution

Register an informer for the console-serving-cert secret so the operator watches it for changes. When the secret's resourceVersion changes, the
operator stamps the new value as a console.openshift.io/serving-cert-secret-version annotation on the console Deployment, which triggers an
automatic pod rollout with the new certificate.

Backport of #1093

Test cases

Scenario 1: Verify secret informer registration

$ oc get secret console-serving-cert -n openshift-console -o jsonpath='{.metadata.resourceVersion}'
36546
$ oc get deployment console -n openshift-console -o jsonpath='{.spec.template.metadata.annotations.console.openshift.io/serving-cert-secret-version}'
36546

Expected: The deployment annotation matches the secret's resourceVersion.

Scenario 2: Automatic pod restart on certificate rotation

Delete the serving cert to simulate rotation

$ oc delete secret console-serving-cert -n openshift-console

Wait for service-ca to regenerate it, then verify

$ oc get secret console-serving-cert -n openshift-console -o jsonpath='{.metadata.resourceVersion}'
38672

$ oc get deployment console -n openshift-console -o jsonpath='{.spec.template.metadata.annotations.console.openshift.io/serving-cert-secret-version}'
38672

$ oc get pods -n openshift-console
NAME READY STATUS RESTARTS AGE
console-75d8bfcdf6-6w2wf 1/1 Running 0 55s <-- redeployed
console-75d8bfcdf6-9zn7z 1/1 Running 0 55s <-- redeployed
downloads-6d94488dc7-rnqts 1/1 Running 0 52m
downloads-6d94488dc7-tb4bz 1/1 Running 0 52m

Expected: The annotation updates to the new resourceVersion and console pods are automatically redeployed. Downloads pods remain untouched.

/assign @Leo6Leo

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@jhadvig
Copy link
Copy Markdown
Member Author

jhadvig commented Apr 27, 2026

/retest

@Leo6Leo
Copy link
Copy Markdown
Contributor

Leo6Leo commented Apr 27, 2026

@coderabbitai review

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Apr 27, 2026

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@pkg/console/subresource/deployment/deployment_test.go`:
- Around line 535-547: The test's args struct declares an unused field
authnConfig; remove the authnConfig field from the type args declaration and any
test case literals that set authnConfig so the struct and its usages no longer
reference it (look for the type args declaration and test case constructions in
deployment_test.go, e.g., where an args literal is created) to satisfy
golangci-lint's unused-field complaint.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: cd6dfe95-1fff-4b4a-8665-b7f90add88f0

📥 Commits

Reviewing files that changed from the base of the PR and between 1bb7f6f and 1532fca.

📒 Files selected for processing (4)
  • pkg/console/operator/operator.go
  • pkg/console/operator/sync_v400.go
  • pkg/console/subresource/deployment/deployment.go
  • pkg/console/subresource/deployment/deployment_test.go

Comment on lines 535 to 547
type args struct {
deployment *appsv1.Deployment
consoleConfigMap *corev1.ConfigMap
serviceCAConfigMap *corev1.ConfigMap
authServerCAConfigMap *corev1.ConfigMap
trustedCAConfigMap *corev1.ConfigMap
oAuthClientSecret *corev1.Secret
sessionSecret *corev1.Secret
proxyConfig *configv1.Proxy
infrastructureConfig *configv1.Infrastructure
authnConfig *configv1.Authentication
deployment *appsv1.Deployment
consoleConfigMap *corev1.ConfigMap
serviceCAConfigMap *corev1.ConfigMap
authServerCAConfigMap *corev1.ConfigMap
trustedCAConfigMap *corev1.ConfigMap
oAuthClientSecret *corev1.Secret
sessionSecret *corev1.Secret
consoleServingCertSecret *corev1.Secret
proxyConfig *configv1.Proxy
infrastructureConfig *configv1.Infrastructure
authnConfig *configv1.Authentication
}
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Remove unused authnConfig field from test args.

authnConfig (Line 546) is never referenced and is flagged by golangci-lint as unused.

🧹 Proposed fix
 type args struct {
 	deployment               *appsv1.Deployment
 	consoleConfigMap         *corev1.ConfigMap
 	serviceCAConfigMap       *corev1.ConfigMap
 	authServerCAConfigMap    *corev1.ConfigMap
 	trustedCAConfigMap       *corev1.ConfigMap
 	oAuthClientSecret        *corev1.Secret
 	sessionSecret            *corev1.Secret
 	consoleServingCertSecret *corev1.Secret
 	proxyConfig              *configv1.Proxy
 	infrastructureConfig     *configv1.Infrastructure
-	authnConfig              *configv1.Authentication
 }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
type args struct {
deployment *appsv1.Deployment
consoleConfigMap *corev1.ConfigMap
serviceCAConfigMap *corev1.ConfigMap
authServerCAConfigMap *corev1.ConfigMap
trustedCAConfigMap *corev1.ConfigMap
oAuthClientSecret *corev1.Secret
sessionSecret *corev1.Secret
proxyConfig *configv1.Proxy
infrastructureConfig *configv1.Infrastructure
authnConfig *configv1.Authentication
deployment *appsv1.Deployment
consoleConfigMap *corev1.ConfigMap
serviceCAConfigMap *corev1.ConfigMap
authServerCAConfigMap *corev1.ConfigMap
trustedCAConfigMap *corev1.ConfigMap
oAuthClientSecret *corev1.Secret
sessionSecret *corev1.Secret
consoleServingCertSecret *corev1.Secret
proxyConfig *configv1.Proxy
infrastructureConfig *configv1.Infrastructure
authnConfig *configv1.Authentication
}
type args struct {
deployment *appsv1.Deployment
consoleConfigMap *corev1.ConfigMap
serviceCAConfigMap *corev1.ConfigMap
authServerCAConfigMap *corev1.ConfigMap
trustedCAConfigMap *corev1.ConfigMap
oAuthClientSecret *corev1.Secret
sessionSecret *corev1.Secret
consoleServingCertSecret *corev1.Secret
proxyConfig *configv1.Proxy
infrastructureConfig *configv1.Infrastructure
}
🧰 Tools
🪛 golangci-lint (2.11.4)

[error] 546-546: field authnConfig is unused

(unused)

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pkg/console/subresource/deployment/deployment_test.go` around lines 535 -
547, The test's args struct declares an unused field authnConfig; remove the
authnConfig field from the type args declaration and any test case literals that
set authnConfig so the struct and its usages no longer reference it (look for
the type args declaration and test case constructions in deployment_test.go,
e.g., where an args literal is created) to satisfy golangci-lint's unused-field
complaint.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit, and it is in the test file, not a big deal tho

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Leo6Leo, agreed — it's a minor cosmetic issue in a test file with no functional impact. Feel free to leave it as-is if it's not worth the churn.

@Leo6Leo
Copy link
Copy Markdown
Contributor

Leo6Leo commented Apr 27, 2026

The tests seem flaky.
/retest
/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Apr 27, 2026
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Apr 27, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jhadvig, Leo6Leo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@jhadvig
Copy link
Copy Markdown
Member Author

jhadvig commented Apr 29, 2026

/retest

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Apr 29, 2026

@jhadvig: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-console 1532fca link true /test e2e-aws-console

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants