Add Litestream backup sidecar to existing StatefulSet

[vcarl]

Context

Part of load balancer architecture work (see PR #228 for full analysis).

Risk: Low | Reward: High | Code Changes: None

Description

Add Litestream sidecar to the existing StatefulSet for continuous SQLite backup to DigitalOcean Spaces.

Why This is Safe

• Read-only operation on SQLite (uses WAL mode)
• Doesn't affect app behavior
• Provides disaster recovery immediately
• Easiest rollback: just remove the sidecar

Tasks

• Create DigitalOcean Spaces bucket for backups
• Add Litestream credentials to modbot-env secret:
  • LITESTREAM_ACCESS_KEY_ID
  • LITESTREAM_SECRET_ACCESS_KEY
• Create cluster/litestream-config.yaml ConfigMap with DO Spaces endpoint
• Add Litestream sidecar container to cluster/deployment.yaml
• Test backup/restore in staging

References

• Litestream docs
• PR Architecture analysis: Load balancing with SQLite constraint #228 (architecture analysis)
• /notes/LOAD_BALANCER_QUICK_REF.md

[vcarl]

Triage

Classification: Enhancement — infrastructure, Tier 1 of the load-balancing cluster (#236).

Summary: Add Litestream sidecar to the existing StatefulSet for continuous SQLite replication to DigitalOcean Spaces. No application code changes required.

Assessment:

• Risk is genuinely low: Litestream operates in WAL-mode read-only alongside SQLite, doesn't affect writes or app behavior. Rollback is trivial (remove the sidecar).
• Reward is high independent of scaling plans: provides disaster recovery capability today. A StatefulSet PVC loss without this would mean total data loss.
• This is the one scaling-cluster issue that makes sense to prioritize even if Implement HTTP service with guild routing #234/Implement Gateway StatefulSet with guild sharding #235 are deferred indefinitely.

Pre-conditions: DigitalOcean Spaces bucket + credentials needed (LITESTREAM_ACCESS_KEY_ID, LITESTREAM_SECRET_ACCESS_KEY). The ConfigMap + sidecar container changes are well-specified in the issue body.

Priority: Medium — valuable as a standalone safety measure, not just a stepping stone to full scaling.

[vcarl]

Triage — github-test

Classification: Enhancement — infrastructure (disaster recovery)
Priority: Medium — valuable standalone, independent of full scaling plans
Complexity: Low (config + YAML only, no application code changes)
Labels: enhancement ✓

Summary

Add a Litestream sidecar to the existing StatefulSet for continuous SQLite replication to DigitalOcean Spaces. This provides disaster recovery today — a PVC loss without this means total data loss.

Why prioritize even if scaling (#235) is deferred

This is the one item in the load-balancing cluster (#236) that has near-term value completely independent of horizontal scaling. A production incident that destroys the StatefulSet PVC would be catastrophic without this in place.

Implementation is well-specified

The issue body has all the steps:

1. Create DO Spaces bucket
2. Add LITESTREAM_ACCESS_KEY_ID + LITESTREAM_SECRET_ACCESS_KEY to modbot-env secret
3. Create cluster/litestream-config.yaml ConfigMap
4. Add Litestream sidecar container to cluster/deployment.yaml
5. Test restore in staging

Rollback is trivial: remove the sidecar container definition.

Dependencies

None — no application code changes, no dependency on #232 (SERVICE_MODE) or any other open issue.

Next step: Whoever has Kubernetes cluster access should be able to implement this in a maintenance window. Recommend scheduling this before any public promotion of the product.