[Phase 4] Network latency and chaos simulation

[cbaugus]

Inject artificial network conditions (latency, packet loss, bandwidth throttling) per node to simulate real-world degradation and test application resilience.

Modes

Artificial request delay

• Add a configurable sleep before each request fires (in addition to the rate-limiter stagger)
• Simulates high-latency regions (e.g. satellite, cross-continent)
• CHAOS_LATENCY_MS=50 — adds 50ms before every request
• CHAOS_LATENCY_JITTER_MS=20 — adds random ±20ms jitter on top

Request drop / simulated packet loss

• Randomly skip sending a percentage of requests (count them as errors)
• CHAOS_DROP_RATE=0.05 — drop 5% of requests before they leave the client
• Dropped requests are recorded as chaos_drop in the error category metric

Per-node chaos (not global)

• Each node can have its own chaos config — useful for simulating one degraded region
• In a 6-node cluster you can make node europe-west1 add 200ms latency while others run clean
• Leader distributes per-node chaos config after Raft election

Config

CHAOS_LATENCY_MS=0         # extra ms before each request (default: 0 = disabled)
CHAOS_LATENCY_JITTER_MS=0  # random jitter range (default: 0)
CHAOS_DROP_RATE=0.0        # fraction of requests to drop (0.0–1.0, default: 0)

Metrics

• rust_loadtest_chaos_requests_dropped_total — counter of artificially dropped requests
• rust_loadtest_chaos_latency_added_ms — histogram of injected latency values
• Both labeled with region so you can isolate chaos effects in Grafana

Notes

• This is client-side simulation, not tc netem / kernel-level — no root/privileged access needed
• Works inside GCP containers without any special permissions
• Combine with real multi-region deployment for realistic degraded-region scenarios

[cbaugus]

Still relevant — no architecture change needed

Network latency and chaos simulation (injecting artificial delays, packet loss simulation) is independent of the consensus layer and applies equally to the new independent-node architecture. Each node can simulate network conditions for its own outbound requests without any coordination.

Keeping open for future implementation.