TechPrep
The trade-offsReadPracticeCheck

Reliability: SLOs, timeouts, retries and regions

Decide how reliable the system must be, then spend that error budget deliberately with timeouts, retries, isolation and graceful degradation.

Start from a service level objective: "99.9% of requests succeed within 300 ms over 28 days" leaves an error budget of about 40 minutes a month. Every design choice either spends or protects that budget.

The standard toolkit

Multi-region

ModeRTO / RPOCost and complexity
Backup and restoreHours / up to the last backupLow
Warm standby (async replica)Minutes / seconds of dataMedium; test failover regularly
Active-activeNear zero / near zero for partitioned dataHigh; conflict handling or strict data partitioning by region
For LLM-backed systems, the model provider is a dependency with its own rate limits and incidents. Plan for it like any other: timeouts suited to long generations, streaming, queueing with backpressure, a fallback model or degraded mode, and per-tenant quotas.

Practice

Written for this site to drill the lesson. Not reported interview questions.

  1. Set SLOs for an internal coding assistant and say what you would page on.
  2. A downstream service has a 1% error rate and you retry 3 times. Estimate the extra load and when it becomes dangerous.
  3. Design graceful degradation for a checkout page when the recommendation and fraud-scoring services are slow.

Check yourself

Roughly how much downtime does 99.9% allow over 28 days?

About 40 minutes.

When should you not retry?

When the operation is not idempotent, or when a retry budget is exhausted during a brownout.

Saved in this browser.