fix(reconcile): refactor pvc reconcillation, add conflict retries

[GrigoryPervakov]

Why

PVC changes should not trigger pod restart and block other replica resources from updating, but they should block updates to the next replica.
Frequent reconciles may face Conflict errors that can be retried.

What

Decouple STS and PVC revision.
Allow STS update if PVC update failed.
Add retries on conflict.

Related Issues

Related to #133, #131

[Copilot]

Pull request overview

This PR refactors replica reconciliation to decouple PVC updates from StatefulSet (STS) spec revisions so that PVC update failures no longer silently block STS updates, and adds conflict retry handling to make updates more resilient under frequent reconciles.

Changes:

• Split STS vs PVC revision tracking: ignore VolumeClaimTemplates in STS revision hashing and introduce a separate PVC revision/hash check.
• Update reconciliation flow so PVC update failures no longer prevent STS updates, while still preventing progression to the next replica.
• Add RetryOnConflict retries to resource updates and expand envtest coverage to better model PVC immutability behavior.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
internal/controllerutil/annotations.go	Adds a helper to consistently set the spec-hash annotation.
internal/controller/testutil/suite.go	Wraps the envtest client with an update interceptor to simulate PVC immutability; enhances STS readiness simulation by creating pods/PVCs.
internal/controller/resources.go	Adds conflict retries to updates; introduces PVC lookup by STS; refactors replica reconciliation to update PVCs without blocking STS updates and to track PVC revision separately.
internal/controller/keeper/templates.go	Excludes VolumeClaimTemplates from the STS revision hash computation.
internal/controller/keeper/templates_test.go	Adds a test ensuring STS revision does not change when the data disk spec changes.
internal/controller/keeper/sync.go	Tracks per-replica PVC state and a separate pvcRevision; includes PVC in diff detection and reconciliation input.
internal/controller/keeper/sync_test.go	Updates expectations around spec-hash annotations and replica state setup for restart-on-config-change behavior.
internal/controller/clickhouse/templates.go	Excludes VolumeClaimTemplates from the STS revision hash computation.
internal/controller/clickhouse/templates_test.go	Adds a test ensuring STS revision does not change when the data disk spec changes.
internal/controller/clickhouse/sync.go	Tracks per-replica PVC state and a separate pvcRevision; includes PVC in diff detection and reconciliation input.
internal/controller/clickhouse/controller_test.go	Adds an integration-style test verifying STS updates proceed even when PVC updates fail, and that reconciliation doesn’t move to the next replica on PVC failure.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.