Replace Terraform ESS provisioning with oblt-cli; add Windows ARM64 CI by michel-laterman · Pull Request #13546 · elastic/elastic-agent

michel-laterman · 2026-04-09T00:00:58Z

Summary

Replace Terraform-based ESS deployment provisioning with oblt-cli for all integration test platforms, removing the Terraform dependency from CI runners
Add Windows ARM64 integration test steps targeting Azure agents
Based on the approach from 🚧 oblt-cli(hosted): support running ITs using oblt-cli #8591 by @v1v, rebased and updated for current main

What does this PR do?

oblt-cli migration (all platforms)

Replaces terraform apply/terraform destroy with oblt-cli cluster create custom --template ess-ea-it and oblt-cli cluster destroy. Credentials are now loaded from GCP Secret Manager via oblt-cli cluster secrets env instead of Terraform outputs or Buildkite metadata.

Scripts changed:

ess.sh / ess.ps1 — new ess_up(), ess_down(), ess_load_secrets() using oblt-cli
ess_start.sh / ess_down.sh — simplified, no Terraform state management
integration_tests_tf.sh / integration_tests_tf.ps1 — use ess_load_secrets instead of Buildkite metadata reads

Pipeline plugins replaced:

Removed: vault_ec_key_prod (EC API key via Vault)
Added: google_oidc_observability_plugin + oblt_cli_plugin + vault_github_token

Removed:

test_infra/ess/ directory (Terraform configs)
terraform from .tool-versions
ASDF_TERRAFORM_VERSION from pipeline env

Windows ARM64 integration tests (new)

Added "Stateful: Windows ARM64" group to bk.integration.pipeline.yml:

Targets Azure agents (provider: azure, vmSize: Standard_D4ps_v6)
Full test matrix matching Windows x86_64 (default, fleet, fleet-endpoint-security, fleet-privileged, standalone-upgrade, upgrade, upgrade-flavor, install-uninstall)
sudo and non-sudo variants

Note: Local mage integration:* commands are not affected — they use separate Go-based provisioners in pkg/testing/ess/.

Prerequisites (separate PRs/repos)

CI images (elastic/ci-agent-images): Build Windows ARM64 image for elastic-agent — update IMAGE_WIN_ARM64 placeholder once ready
GCP permissions (elastic/oblt-infra): Verify elastic-agent pipeline principal has google-oblt-cluster-secrets-access role

Test plan

/test extended to verify ESS stack provisions correctly via oblt-cli
Verify existing Linux/Windows x86 tests pass (no regression)
Verify ESS cleanup step runs successfully
Verify retry path works (oblt-cli creates new cluster on retry)
Windows ARM64 steps execute once CI image is available

🤖 Generated with Claude Code

mergify · 2026-04-09T00:01:46Z

This pull request does not have a backport label. Could you fix it @michel-laterman? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

@v1v

…4 CI Replace the Terraform-based ESS deployment provisioning with oblt-cli for all integration test platforms. This removes the Terraform dependency from CI runners, enabling support for Windows ARM64 where Terraform has no native binary. Key changes: - ess.sh/ess.ps1: Use `oblt-cli cluster create custom --template ess-ea-it` instead of `terraform apply`. Credentials are now loaded from GCP Secret Manager via `oblt-cli cluster secrets env` rather than Terraform outputs. - Pipeline plugins: Replace `vault_ec_key_prod` (EC API key) with `google_oidc_observability_plugin` + `oblt_cli_plugin` + `vault_github_token` - Add "Stateful: Windows ARM64" test group targeting Azure agents with the full test matrix matching Windows x86_64 - Remove `test_infra/ess/` Terraform configs and Terraform from .tool-versions Based on the approach from elastic#8591 by @v1v. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

v1v · 2026-04-14T13:32:02Z

@@ -7,25 +7,20 @@ env:
  IMAGE_UBUNTU_2404_X86_64: "platform-ingest-elastic-agent-ubuntu-2404-1772525581"


AFAIK, we don't support FIPS in oblt-cli yet

v1v · 2026-04-14T14:32:57Z

+  # Source the secrets file
+  # shellcheck source=/dev/null
+  local src_rc=0
+  source "${secrets_file}" || src_rc=$?


for some reason the source is not loading the variables :/

I bet, we need to do some export of the file

See 10b8488 (this PR)

v1v · 2026-04-14T18:38:32Z

-  $Env:KIBANA_HOST = & terraform output -raw kibana_endpoint
-  $Env:KIBANA_USERNAME = $Env:ELASTICSEARCH_USERNAME
-  $Env:KIBANA_PASSWORD = $Env:ELASTICSEARCH_PASSWORD
-  $Env:INTEGRATIONS_SERVER_HOST = & terraform output -raw integrations_server_endpoint


What's this? Is fleet-server?

with oblt-cli there are different URLs:

KIBANA_FLEET_HOST=https://bob.kb.us-west2.gcp.elastic-cloud.com:443

ELASTICSEARCH_HOST=https://ea-hosted-it-ess-ea--hnejl.es.us-west2.gcp.elastic-cloud.com:443

KIBANA_FLEET_HOST=https://ea-hosted-it-ess-ea--hnejl.kb.us-west2.gcp.elastic-cloud.com:443

FLEET_URL=https://ea-hosted-it-ess-ea--hnejl.fleet.us-west2.gcp.elastic-cloud.com:443

INGEST_URL=https://ea-hosted-it-ess-ea--hnejl.ingest.us-west2.gcp.elastic-cloud.com:443

ELASTIC_APM_SERVER_URL=https://ea-hosted-it-ess-ea--hnejl.apm.us-west2.gcp.elastic-cloud.com:443

Need to find the correct URL so we can replace INTEGRATIONS_SERVER_HOST with the relevant env variable provided by oblt-cli

I think it's ELASTIC_APM_SERVER_URL

See 14559fe

github-actions · 2026-04-14T18:46:38Z

🔍 Preview links for changed docs

docs/test-framework-dev-guide.md

v1v · 2026-04-14T20:13:02Z

-  $Env:KIBANA_PASSWORD = $Env:ELASTICSEARCH_PASSWORD
-  $Env:INTEGRATIONS_SERVER_HOST = & terraform output -raw integrations_server_endpoint
-  Pop-Location
+  & oblt-cli cluster create custom `


Let's wait for elastic/oblt-cli-buildkite-plugin#41 so we can add support for Windows too

Wonder if we can split this PR into three steps:

Linux support

Windows support

FIPS support

For Linux I think it should work as is, for Windows we need a few changes to support that platform in the existing BK plugin, and for FIPS, it will not be available in the near future.

I think we can use the FIPS image in the non frh-staging environment

…l-laterman/elastic-agent into feature/oblt-cli-win-arm64 * 'feature/oblt-cli-win-arm64' of https://github.com/michel-laterman/elastic-agent: [main][Automation] Update elastic/beats to 323f731f84e5 (elastic#13622) Update OTel Collector components to v0.149.0/v1.55.0 (elastic#13599) build(deps): bump peter-evans/create-pull-request from 8.1.0 to 8.1.1 (elastic#13593) Fix clean stopping of beats on windows (elastic#13581) Handle multiple values for fleet and kibana headers (elastic#13506) [otel/kube-stack] Update the OpenTelemetry Kube Stack Helm Chart to latest versions (elastic#13522) Automate updates to opentelemetry-collector-components (elastic#13578) [beatreceiver] Add integration testing for kafka output (elastic#13550) Bump the version on main to 9.5.0 (elastic#13532) [beatreceivers] Enable otel runtime by default for kafka and logstash output (elastic#13514) Add Elastic Agent 9.3.3+IAR release notes (elastic#13556) fix(flake): clean up prefix install data dirs in RPM test fixture (elastic#13552) [Automation] Bump Golang version to 1.25.9 (elastic#13559) feat(helm): add support for Fleet URL and token from Kubernetes Secret (elastic#13507)

elasticmachine · 2026-04-15T13:34:57Z

💔 Build Failed

Buildkite Build
Commit: cc9bd08

Failed CI Steps

History

💔 Build #37230 failed 7f89280
💔 Build #37228 failed 10b8488
💔 Build #37001 failed a986aff
💔 Build #36996 failed 859eff5

cc @michel-laterman

mergify bot assigned michel-laterman Apr 9, 2026

michel-laterman force-pushed the feature/oblt-cli-win-arm64 branch from 46c3784 to 859eff5 Compare April 9, 2026 18:55

macroscopeapp bot reviewed Apr 9, 2026

View reviewed changes

Comment thread .buildkite/scripts/steps/ess.sh

Comment thread .buildkite/scripts/steps/ess.sh

michel-laterman and others added 2 commits April 9, 2026 14:34

macroscope feedback

a986aff

michel-laterman force-pushed the feature/oblt-cli-win-arm64 branch from 859eff5 to a986aff Compare April 9, 2026 21:34

macroscopeapp bot reviewed Apr 9, 2026

View reviewed changes

Comment thread .buildkite/scripts/steps/ess.sh Outdated

github-actions bot mentioned this pull request Apr 13, 2026

[agent-suggestions] Agent suggestions - 2026-04-13 elastic/ai-github-actions-playground#2920

Open

4 tasks

v1v reviewed Apr 14, 2026

View reviewed changes

Apply suggestion from @v1v

b56bcbe

v1v reviewed Apr 14, 2026

View reviewed changes

v1v added 2 commits April 14, 2026 20:42

use ELASTIC_APM_SERVER_URL

14559fe

support exporting from source

10b8488

github-actions bot deployed to docs-preview April 14, 2026 18:46 View deployment

Merge branch 'main' into feature/oblt-cli-win-arm64

7f89280

github-actions bot deployed to docs-preview April 14, 2026 19:28 View deployment

v1v reviewed Apr 14, 2026

View reviewed changes

v1v added 2 commits April 15, 2026 14:33

support a new version with windows support

6c712ac

github-actions bot deployed to docs-preview April 15, 2026 13:15 View deployment

v1v mentioned this pull request Apr 15, 2026

Replace Terraform ESS provisioning with oblt-cli for Windows and Linux but no FIPS #13642

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace Terraform ESS provisioning with oblt-cli; add Windows ARM64 CI#13546

Replace Terraform ESS provisioning with oblt-cli; add Windows ARM64 CI#13546
michel-laterman wants to merge 8 commits intoelastic:mainfrom
michel-laterman:feature/oblt-cli-win-arm64

michel-laterman commented Apr 9, 2026

Uh oh!

mergify bot commented Apr 9, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

v1v Apr 14, 2026

Uh oh!

v1v Apr 14, 2026

Uh oh!

v1v Apr 14, 2026

Uh oh!

v1v Apr 14, 2026 •

edited

Loading

Uh oh!

v1v Apr 14, 2026

Uh oh!

github-actions bot commented Apr 14, 2026 •

edited

Loading

Uh oh!

v1v Apr 14, 2026

Uh oh!

v1v Apr 14, 2026

Uh oh!

michel-laterman Apr 15, 2026

Uh oh!

elasticmachine commented Apr 15, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -7,25 +7,20 @@ env:
		IMAGE_UBUNTU_2404_X86_64: "platform-ingest-elastic-agent-ubuntu-2404-1772525581"

Conversation

michel-laterman commented Apr 9, 2026

Summary

What does this PR do?

oblt-cli migration (all platforms)

Windows ARM64 integration tests (new)

Prerequisites (separate PRs/repos)

Test plan

Uh oh!

mergify bot commented Apr 9, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

v1v Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

v1v Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

v1v Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

v1v Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

v1v Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Uh oh!

v1v Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

v1v Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

michel-laterman Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

v1v Apr 14, 2026 •

edited

Loading

github-actions bot commented Apr 14, 2026 •

edited

Loading

elasticmachine commented Apr 15, 2026 •

edited

Loading