Kubedog based tracking issue with atomic/rollback-on-failure helm flags

[r9odt]

Operating system

Fedora Linux 43

Helmfile Version

1.4.0

Helm Version

v4.1.1

Bug description

When using a helmfile with the wait and atomic parameters set to true, tracking begins after the release installation completes. If created pods enter the Error or CrashLoopBackOff status during the installation, Helm aborts the installation after the configured timeout and tracking does not seem to start.

The helm documentation states that the wait flag is automatically set when using the --atomic or --rollback-on-failure flags.
--atomic if set, the installation process deletes the installation upon failure. The --wait flag will be set automatically if --atomic is used.
https://helm.sh/docs/v3/helm/helm_install

--rollback-on-failure if set, Helm will rollback (uninstall) the installation upon failure. The --wait flag will default to "watcher" if --rollback-on-failure is set.
https://helm.sh/docs/helm/helm_install/

When using atomic flags, I expected to be able to track the state of created Pods before Helm performed a rollback. This would allow to view deployment logs and make tracking more informative when errors occur in the application.

To clarify, if i don't specify the wait and atomic parameters in the helmfile, kubedog tracking works, but the release is updated and the new created Pod remains in the CrashLoopBackOff state.

Example helmfile.yaml

helmDefaults:
  createNamespace: false
  atomic: true
  wait: true
  historyMax: 1
  kubeContext: "{{ .StateValues.kubeContext }}"

Error message you've seen (if any)

EXIT STATUS
  1

STDERR:
  Flag --atomic has been deprecated, use --rollback-on-failure instead
  level=WARN msg="upgrade failed" name=myrelease error="resource Deployment/testns/myrelease-api not ready. status: InProgress, message: Pending termination: 1\ncontext deadline exceeded"
  Error: UPGRADE FAILED: release myrelease failed, and has been rolled back due to rollback-on-failure being set: resource Deployment/testns/myrelease-api not ready. status: InProgress, message: Pending termination: 1
  context deadline exceeded

Steps to reproduce

use atomic (optionally with wait) parameters in helmDefaults with true value

Working Helmfile Version

1.3.2 but it does not have the kubedog based tracker

Relevant discussion

No response

[yxxhero]

@r9odt do you like this feature? it's first version now. need you feedback.

[r9odt]

@r9odt do you like this feature? it's first version now. need you feedback.

@yxxhero

The first impression is positive.

I tested it on a small application, one deployment-one replica. When I tried deploying the application with wait and atomic disabled, kubedog started after deployment, and I saw status changes and, at first error, got a pod error log right at the start.

But currently i use atomic: true in almost every scenario, which prevents me from fully utilizing this feature.

It seems like this would be useful for some kinds of CI scenarios when a deployment may fails. This way, it can display log from the pod(s) to the user without additional processes and CI wrappers or re-running the deployment to get logs via kubectl logs or other ways.
Also i haven't tested cases where pod stuck in Pending (e.g. -- an invalid nodeSelector is specified) but outputting events for this cases could also be useful if kubedog supports it.

[yxxhero]

Root Cause Analysis

Thank you for reporting this issue! Your understanding is correct.

When `atomic: true` is enabled, the rollback behavior of the `--atomic` flag is executed entirely within Helm's internal process:

1. Helm executes upgrade and waits for resources to become ready
2. If deployment fails (e.g., Pods enter CrashLoopBackOff), Helm automatically rolls back
3. Then it returns an error to helmfile

This means:

• Rollback happens before helmfile can intervene
• helmfile cannot start kubedog tracking before Helm rolls back
• Users cannot see logs and error information during deployment

Solution

We are working on a fix for this issue. The approach is:

When `trackMode: kubedog` is enabled with atomic semantics:

1. Do not pass `--atomic` flag to Helm (skip Helm's internal wait and rollback)
2. Let Helm execute upgrade normally (without waiting)
3. helmfile immediately starts kubedog to track resources (users can see logs and errors)
4. If kubedog tracking fails, helmfile manually executes `helm rollback`

This preserves the atomic semantics (rollback on failure) while allowing kubedog to track resources during deployment.

Workaround

Before the fix is released, you can use:

```yaml
helmDefaults:
atomic: false # Disable atomic
wait: false # Disable wait
trackMode: kubedog # Enable kubedog
```

This allows kubedog to work properly, but failed releases won't automatically roll back. You'll need to manually rollback failed releases.

We're actively implementing this fix and expect to release it soon.

[yxxhero]

@r9odt WDYT?

[r9odt]

@yxxhero Sorry for the delay.
The analysis provided is correct, and the description of the solution approach also seems correct. To provide more details, I need to compile a helmfile from the PR and test it on my own examples. I will try to do it today, at the latest tomorrow (04 March 2026-05 March /2026)

When we disable atomic and use kubedog, what rollback timeout will be used in the helmfile?

Also, I should highlight cases with a Pending pod state, since the atomic flag also tells Helm to rollback after the deployment timeout.

[pavelsocolov]

@r9odt
Your problem is related to my bug.
Any idea when you can take a look? With the "atomic" parameter enabled, kubedog logging doesn't work. Without "atomic," it's difficult to work.

#2445

[r9odt]

I have check build from PR #2450 and get the next results.

helmDefaults:
  atomic: true
  trackMode: kubedog
  wait: true

1. The first install isn't removed when an error occurs.

STDERR:
  Error: release has no 0 version

2. When a pod is in the crashloopbackoff state, the Helm release is rolled back after deploy timeout reached (timeout in helmDefaults). Everything appears correct here.
3. When a pod is in the pending state, the Helm release is rolled back after track timeout (~5m)
4. Pressing Ctrl-C doesn't stop tracking and sometimes causes panic. It also doesn't seem to roll back after interruption, but I'm not sure.

Rolling back release testapi due to kubedog tracking failure (atomic mode enabled)
Rolling back testapi
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x36103e8]

goroutine 104 [running]:
os.(*Process).signal(0x300360db70c0?, {0x95f9d28?, 0xdb49768?})
        go/src/os/exec_unix.go:84 +0x28
os.(*Process).Signal(...)
        go/src/os/exec.go:352
github.com/helmfile/helmfile/pkg/helmexec.Output({0x96259c8, 0x30035ff700a0}, 0x300360a5e4e0, 0x1, {0x300360db7190, 0x1, 0x30035f9ff008?})
        helmfile/pkg/helmexec/runner.go:103 +0xbb0
github.com/helmfile/helmfile/pkg/helmexec.ShellRunner.Execute({{0x0, 0x0}, 0x1, 0x3003603a2160, {0x96259c8, 0x30035ff700a0}}, {0x90ccd71?, 0x300360db7240?}, {0x30036142c300, 0x6, ...}, ...)
        helmfile/pkg/helmexec/runner.go:44 +0x173
github.com/helmfile/helmfile/pkg/helmexec.(*execer).exec(0x300360c20160, {0x3003615e2140?, 0x0?, 0x3003603a2160?}, 0x300360d20f00, 0x0)
        helmfile/pkg/helmexec/exec.go:942 +0x483
github.com/helmfile/helmfile/pkg/helmexec.(*execer).RollbackRelease(0x300360c20160, {0xe5fd900?, 0x9222c63?, {0x4d?, 0x300360db74d8?}}, {0x300360240370, 0x10}, {0x300361141080, 0x2, 0x2})
        helmfile/pkg/helmexec/exec.go:844 +0x1bf
github.com/helmfile/helmfile/pkg/state.(*HelmState).handleKubedogFailure(0x30036058ea88, {0x967dc01?, 0x918b2af?, {0x0?, 0x0?}}, {0x967dc38, 0x300360c20160}, 0x3003616cd688, {0x9591e60, 0x300361141060}, ...)
        helmfile/pkg/state/state.go:1285 +0x18b
github.com/helmfile/helmfile/pkg/state.(*HelmState).SyncReleases.func3(0x1)
        helmfile/pkg/state/state.go:1158 +0xeac
github.com/helmfile/helmfile/pkg/state.(*HelmState).scatterGather.func1(0x30035fc00a80?)
        helmfile/pkg/state/state_run.go:33 +0x22
created by github.com/helmfile/helmfile/pkg/state.(*HelmState).scatterGather in goroutine 28
        helmfile/pkg/state/state_run.go:32 +0x71

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.