Skip to content

Add vSphere disconnected Windows Containers periodic jobs for 4.18-4.22#76411

Open
rrasouli wants to merge 1 commit intoopenshift:mainfrom
rrasouli:add-vsphere-disconnected-winc-periodic-jobs
Open

Add vSphere disconnected Windows Containers periodic jobs for 4.18-4.22#76411
rrasouli wants to merge 1 commit intoopenshift:mainfrom
rrasouli:add-vsphere-disconnected-winc-periodic-jobs

Conversation

@rrasouli
Copy link
Contributor

@rrasouli rrasouli commented Mar 17, 2026

Summary

This PR adds periodic CI jobs for Windows Containers testing in disconnected (air-gapped) vSphere environments for OpenShift releases 4.18 through 4.22.

This supports the migration from Jenkins to Prow for Windows Containers functionality testing (JIRA: WINC-1611).

Changes

New Periodic Jobs

Added vsphere-ipi-disconnected-ovn-winc-f28 periodic job to each release:

  • Schedule: Monthly on the 15th at 8:00 AM UTC (cron: 0 8 15 * *)
  • Cluster Profile: vsphere-dis-2 (disconnected vSphere)
  • Test Scenario: Windows_Containers
  • Releases: 4.18, 4.19, 4.20, 4.21, 4.22

New Workflow and Steps

Includes workflow and supporting steps from Weinan's PR #76193:

  • Workflow: cucushift-installer-rehearse-vsphere-ipi-disconnected-ovn-winc
  • Provision chain: Sets up bastion host, mirrors images, configures OVN hybrid, installs WMCO, creates Windows MachineSets
  • Deprovision chain: Cleans up cluster and bastion resources
  • Image mirroring steps:
    • mirror-images-by-oc-adm-via-tunnel - Mirrors via SSH tunnel to bastion
    • mirror-images-by-oc-image-in-bastion - Mirrors using oc-image tool on bastion

Bug Fix

Includes fix from Weinan's PR #76362:

  • Adds missing ipi-install-vsphere-registry step to vSphere OVN WINC provision chain
  • Fixes registry configuration issue that prevented dockercfg secrets from being created

Testing Approach

Windows Worker Type: ✅ MachineSet-only (NO BYOH)

  • Uses ipi-conf-vsphere-windows-machineset step
  • Removed windows-conf-operator step (which creates SSH keys for BYOH)
  • Pure MachineSet-based Windows node deployment
  • No BYOH provisioning, no SSH configuration

Disconnected Setup:

  1. Provision bastion host in connected network
  2. Mirror required images to bastion registry
  3. Install cluster in disconnected vSphere environment
  4. Configure mirror registry DNS
  5. Deploy WMCO operator via Operator Lifecycle Manager
  6. Create Windows MachineSets (WMCO automatically configures nodes)
  7. Run QE Windows workload tests

Why MachineSet-only:

  • Simpler, more standard deployment path
  • Tests production-like Windows node configuration
  • BYOH can be tested separately if needed via dedicated jobs
  • Aligns with connected vSphere WINC jobs

Files Modified

Configuration (5 files):

  • ci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.{18,19,20,21,22}__amd64-nightly.yaml

Generated Jobs (5 files):

  • ci-operator/jobs/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.{18,19,20,21,22}-periodics.yaml

Step Registry (19 new files):

  • Workflow, provision/deprovision chains, and image mirroring steps

Total: 29 files changed, 1043 insertions(+), 12 deletions(-)

Relationship to Other PRs

Testing Plan

  1. Wait for PR to merge
  2. Monitor first periodic execution on 2026-04-15
  3. Verify disconnected cluster provisioning succeeds
  4. Verify Windows MachineSets are created (not BYOH nodes)
  5. Verify Windows workload tests execute
  6. Compare results with Jenkins functionality-testing jobs

Test Results from Manual Run

Job executed successfully:

  • 22 tests passed
  • 10 tests failed (mostly flaky Disruptive/Slow tests)
  • 8 tests skipped
  • Runtime: 1h38m29s
  • Infrastructure: Working correctly
  • Windows nodes: Created via MachineSet

JIRA

https://issues.redhat.com/browse/WINC-1611

Related Jenkins Configuration

Migrating from: flexy-templates/functionality-testing/aos-4_22/hosts/windows_workers/

  • Script: scaleup_windows_workers.sh
  • Mirror script: mirror_windows_container_images.sh
  • Note: Jenkins uses BYOH optionally; this Prow job uses MachineSet only

@openshift-ci openshift-ci bot requested review from jechen0648 and jhou1 March 17, 2026 16:46
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 17, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rrasouli
Once this PR has been reviewed and has the lgtm label, please assign jinyunma, xia-zhao-rh for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@rrasouli
Copy link
Contributor Author

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-vsphere-ipi-ovn-winc-f7

@openshift-ci-robot
Copy link
Contributor

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@rrasouli rrasouli force-pushed the add-vsphere-disconnected-winc-periodic-jobs branch 2 times, most recently from 94526cc to 306836b Compare March 18, 2026 07:00
@rrasouli
Copy link
Contributor Author

Correction: windows-conf-operator Step Restored

Update: The windows-conf-operator step has been restored to the provision chain.

Why This Step Is Required

The windows-conf-operator step creates the cloud-private-key secret containing the SSH private key that WMCO uses to configure ALL Windows nodes, regardless of whether they are:

  • ✅ MachineSet-created nodes
  • ✅ BYOH nodes

How WMCO Works

  1. MachineSet creates Windows VM (or BYOH provides existing VM)
  2. SSH key is injected into the Windows instance during provisioning
  3. WMCO SSH into the node using the cloud-private-key secret
  4. WMCO configures the node: Installs kubelet, CNI plugins, joins cluster

Evidence

All other WINC provision chains use this step:

  • cucushift-installer-rehearse-aws-ipi-ovn-winc-provision
  • cucushift-installer-rehearse-azure-ipi-ovn-winc-provision
  • cucushift-installer-rehearse-gcp-ipi-ovn-winc-provision
  • cucushift-installer-rehearse-nutanix-ipi-ovn-winc-provision
  • cucushift-installer-rehearse-vsphere-ipi-ovn-winc-provision

Final Provision Chain Order

- ref: ssh-bastion                         # Enable SSH to bastion
- ref: windows-conf-operator               # Create SSH secret for WMCO ← REQUIRED
- ref: ipi-conf-vsphere-windows-machineset # Create Windows MachineSets
- ref: cucushift-winc-prepare              # Deploy test workloads

The workflow now correctly matches the pattern used by all other WINC jobs.

@rrasouli
Copy link
Contributor Author

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-vsphere-ipi-disconnected-ovn-winc-f28

@openshift-ci-robot
Copy link
Contributor

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

This PR adds periodic CI jobs for Windows Containers testing in disconnected
(air-gapped) vSphere environments for OpenShift 4.18 through 4.22.

Changes:
- Add vSphere disconnected OVN WINC workflow and supporting steps
- Add periodic job vsphere-ipi-disconnected-ovn-winc-f28 to each release
- Include Weinan's workflow from PR openshift#76193 (provision/deprovision chains)
- Include Weinan's PR openshift#76362 fix (add ipi-install-vsphere-registry step)
- Add mirror-images-by-oc-adm-via-tunnel and mirror-images-by-oc-image-in-bastion steps
- FIX: Remove squid proxy usage in mirror-images-by-oc-image-in-bastion (proxy not running)

Job configuration:
- Runs monthly (15th of each month at 8:00 AM UTC)
- Uses vsphere-dis-2 cluster profile
- Tests Windows Containers scenario with disconnected registry
- Uses MachineSet-based Windows workers (WMCO configures via SSH)

Workflow includes:
- Bastion host provisioning for disconnected network
- Image mirroring via oc-image tool (direct connection, no proxy)
- OVN hybrid networking configuration
- WMCO operator deployment
- windows-conf-operator: Creates SSH secret for WMCO to configure Windows nodes
- Windows MachineSet creation
- QE Windows workload preparation

Bug fix: mirror-images-by-oc-image-in-bastion was trying to use squid proxy
at 127.0.0.1:3128 which doesn't exist on bastion, causing connection refused.
Fixed by removing proxy env vars - bastion can reach CI registry directly.

JIRA: WINC-1611
@rrasouli
Copy link
Contributor Author

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-vsphere-ipi-disconnected-ovn-winc-f28

@openshift-ci-robot
Copy link
Contributor

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@rrasouli
Copy link
Contributor Author

/pj-rehearse abort

@openshift-ci-robot
Copy link
Contributor

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@rrasouli rrasouli force-pushed the add-vsphere-disconnected-winc-periodic-jobs branch from 306836b to dfafdd4 Compare March 18, 2026 07:48
@openshift-ci-robot
Copy link
Contributor

[REHEARSALNOTIFIER]
@rrasouli: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-openshift-openshift-tests-private-main-debug-winc-vsphere-ipi openshift/openshift-tests-private presubmit Registry content changed
pull-ci-openshift-openshift-tests-private-release-5.0-debug-winc-vsphere-ipi openshift/openshift-tests-private presubmit Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.23-debug-winc-vsphere-ipi openshift/openshift-tests-private presubmit Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.22-debug-winc-vsphere-ipi openshift/openshift-tests-private presubmit Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.21-debug-winc-vsphere-ipi openshift/openshift-tests-private presubmit Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.20-debug-winc-vsphere-ipi openshift/openshift-tests-private presubmit Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.19-debug-winc-vsphere-ipi openshift/openshift-tests-private presubmit Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.18-debug-winc-vsphere-ipi openshift/openshift-tests-private presubmit Registry content changed
pull-ci-openshift-openshift-tests-private-release-4.16-debug-winc-vsphere-ipi openshift/openshift-tests-private presubmit Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.18-amd64-nightly-azure-ipi-disc-oidc-oc-mirror-f28-destructive N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-4.18-amd64-nightly-gcp-ipi-disc-priv-oidc-oc-mirror-fips-f28-destructive N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-vsphere-ipi-ovn-winc-f7 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.18-amd64-nightly-vsphere-ipi-ovn-winc-f14 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-vsphere-ipi-disconnected-ovn-winc-f28 N/A periodic Periodic changed
periodic-ci-openshift-openshift-tests-private-release-4.19-amd64-nightly-ibmcloud-ipi-disc-priv-oc-mirror-f28-destructive N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-vsphere-ipi-ovn-winc-f7 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.23-amd64-nightly-vsphere-ipi-ovn-winc-f7 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.20-amd64-nightly-aws-ipi-disc-priv-sts-ep-oc-mirror-fips-f28-destructive N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-4.20-amd64-nightly-azure-ipi-disc-oidc-oc-mirror-mini-perm-f28-destructive N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-4.19-amd64-nightly-gcp-ipi-disc-priv-oidc-oc-mirror-fips-f28-destructive N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-4.20-amd64-nightly-ibmcloud-ipi-disc-priv-oc-mirror-f28-destructive N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-4.18-amd64-nightly-vsphere-ipi-disconnected-ovn-winc-f28 N/A periodic Periodic changed
periodic-ci-openshift-openshift-tests-private-release-4.20-amd64-nightly-gcp-ipi-disc-priv-oidc-oc-mirror-fips-f28-destructive N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-4.17-amd64-nightly-vsphere-ipi-ovn-winc-f14 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.20-amd64-nightly-vsphere-ipi-disconnected-ovn-winc-f28 N/A periodic Periodic changed

A total of 35 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 18, 2026

@rrasouli: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-vsphere-ipi-disconnected-ovn-winc-f28 306836b link unknown /pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-vsphere-ipi-disconnected-ovn-winc-f28

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@rrasouli
Copy link
Contributor Author

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-vsphere-ipi-disconnected-ovn-winc-f28

@rrasouli
Copy link
Contributor Author

/hold

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 18, 2026
@rrasouli
Copy link
Contributor Author

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-vsphere-ipi-disconnected-ovn-winc-f28

@openshift-ci-robot
Copy link
Contributor

@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants