ian/adding_k8_backend

[ianhodge]

Summary

This PR adds a Kubernetes execution backend to oz-agent-worker and includes the deployment and hardening work needed to make that backend practical to run in a customer Kubernetes environment.

At a high level, the worker can now execute tasks by creating Kubernetes Jobs instead of running them via Docker or the direct backend. The PR also adds a namespace-scoped Helm chart, updates the docs for customer deployment, and tightens the production path with CI coverage, safer chart defaults, and runtime/container hardening.

What changed

Kubernetes backend

• added a new Kubernetes backend implementation in internal/worker/kubernetes.go
• added config parsing / merge support for backend.kubernetes.* in internal/config/config.go and main.go
• execute each task as a Kubernetes Job / Pod in a target namespace
• support setup and teardown hooks for task execution
• propagate configured environment variables into task jobs
• support Kubernetes-specific execution settings including:
  • namespace / kubeconfig selection
  • image pull secret / image pull policy
  • task job service account
  • node selectors / tolerations / resource requests and limits
  • extra labels / annotations
  • active deadline / termination grace period / workspace size limit
  • configurable unschedulable timeout
  • configurable startup preflight_image
• added a startup dry-run Job preflight so policy / RBAC / admission issues surface before task execution begins
• removed the need for a cluster-scoped namespace read at startup; validation stays namespaced
• added stable hash-based labels and job naming to avoid selector collisions after sanitization
• updated worker shutdown cleanup to use a fresh context for backend cleanup

Helm chart

• added a namespace-scoped chart at charts/oz-agent-worker
• chart deploys:
  • long-lived worker Deployment
  • ServiceAccount
  • namespaced Role / RoleBinding
  • worker config ConfigMap
  • optional API key Secret
• chart is designed for in-cluster auth by default
• chart distinguishes between:
  • the worker Deployment service account
  • the optional task Job service account configured via backend.kubernetes.service_account
• chart now requires an explicit image.tag so installs pin a worker image rather than defaulting to latest
• chart defaults the long-lived worker Deployment to a non-root security context with conservative resource requests
• added kubernetesBackend.preflightImage so restricted clusters can override the startup preflight image

CI / packaging / docs

• updated CI to:
  • use the Go version from go.mod
  • run go test ./...
  • lint and render the Helm chart in CI
• fixed .gitignore so the top-level binary is ignored without accidentally ignoring charts/oz-agent-worker/**
• hardened the runtime Dockerfile to run the worker as a non-root user on a pinned Alpine base image
• expanded README.md with:
  • Kubernetes backend configuration and caveats
  • Helm installation flow
  • production notes for explicit image pinning
  • non-root worker defaults
  • preflight image override guidance
  • the distinction between worker and task-job service accounts

Operational notes

• this backend does not require CRDs
• this backend does not create cluster-scoped RBAC resources
• the worker Deployment is intended to run long-term in-cluster
• each task is executed as a Kubernetes Job
• the worker Deployment defaults to non-root, but the task namespace must still allow creating Jobs with a root init container because sidecar materialization currently depends on that pattern
• keep replicaCount=1 for a given worker.workerId; scale by creating multiple releases with distinct worker IDs instead of scaling a single release horizontally
• if cluster policy restricts allowed registries/images, set preflight_image / kubernetesBackend.preflightImage to an allowlisted image

Validation

• gofmt -w on modified Go files
• go test ./...
• go build ./...
• helm lint charts/oz-agent-worker --set worker.workerId=my-worker --set image.tag=v1.2.3
• helm template oz-agent-worker charts/oz-agent-worker --namespace agents --set worker.workerId=my-worker --set image.tag=v1.2.3
• helm lint + helm template again with richer override values to exercise optional chart branches including secret creation, annotations, node selectors, tolerations, resources, setup/teardown hooks, environment entries, and kubernetesBackend.preflightImage
• docker build to verify the hardened runtime image still builds successfully

Reviewer notes

The highest-risk / highest-value areas to review are:

• internal/worker/kubernetes.go for job lifecycle, startup preflight behavior, and failure detection
• main.go + internal/config/config.go for config merge / validation behavior
• charts/oz-agent-worker/* for install ergonomics and namespaced deployment assumptions
• README.md for customer-facing deployment guidance and caveats

Artifacts

• Conversation: https://staging.warp.dev/conversation/5929183c-8ee2-4efd-85c2-1b1f25d71747
• Plan: https://staging.warp.dev/drive/notebook/YX1kOJGSO6gwPPoL4sdv56

Co-Authored-By: Oz oz-agent@warp.dev

[ianhodge]

• ian/adding_k8_backend #36 👈 (View in Graphite)
• Adding customizable idle-on-complete for self hosted worker  #35
• Adding concurrency limit to the self hosted worker config  #34
• Adding a direct backend option of the self hosted runner  #33
• Adding file config support for docker backend #32
• Creating backend abstraction for self hosted worker  #29
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[ianhodge]

These changes are needed for the helm chart