Reliability: SLOs, timeouts, retries and regions
Decide how reliable the system must be, then spend that error budget deliberately with timeouts, retries, isolation and graceful degradation.
Start from a service level objective: "99.9% of requests succeed within 300 ms over 28 days" leaves an error budget of about 40 minutes a month. Every design choice either spends or protects that budget.
The standard toolkit
- Timeouts on every network call, shorter than the caller's own deadline. Propagate deadlines.
- Retries with exponential backoff and jitter, only for idempotent operations, with a retry budget so a brownout does not become a retry storm.
- Circuit breakers and bulkheads so one slow dependency cannot exhaust every thread or connection.
- Load shedding and graceful degradation: drop optional work first (recommendations, enrichment) and keep the core flow alive.
- Cells: partition customers into independent stacks so a bad deploy or poison tenant hurts one cell. Common in multi-tenant SaaS and AI platforms.
Multi-region
| Mode | RTO / RPO | Cost and complexity |
|---|---|---|
| Backup and restore | Hours / up to the last backup | Low |
| Warm standby (async replica) | Minutes / seconds of data | Medium; test failover regularly |
| Active-active | Near zero / near zero for partitioned data | High; conflict handling or strict data partitioning by region |
For LLM-backed systems, the model provider is a dependency with its own rate limits and incidents. Plan for it like any other: timeouts suited to long generations, streaming, queueing with backpressure, a fallback model or degraded mode, and per-tenant quotas.
Practice
Written for this site to drill the lesson. Not reported interview questions.
- Set SLOs for an internal coding assistant and say what you would page on.
- A downstream service has a 1% error rate and you retry 3 times. Estimate the extra load and when it becomes dangerous.
- Design graceful degradation for a checkout page when the recommendation and fraud-scoring services are slow.
Check yourself
Roughly how much downtime does 99.9% allow over 28 days?
About 40 minutes.
When should you not retry?
When the operation is not idempotent, or when a retry budget is exhausted during a brownout.
Saved in this browser.