Caching and its failure modes
Caches are the cheapest latency win and the most common source of subtle bugs: plan invalidation, stampedes and cold starts up front.
| Pattern | How it works | Good for | Risk |
|---|---|---|---|
| Cache-aside | App reads cache, on miss reads DB and fills cache | General read-heavy data | Stale data until TTL or explicit invalidation |
| Write-through | Writes go to cache and DB together | Read-after-write consistency | Write latency; cache full of cold data |
| Write-behind | Write to cache, flush to DB asynchronously | Bursty writes | Data loss if the cache dies before flushing |
| Read-through / CDN | Cache sits in front and fetches on miss | Static and semi-static content | Invalidation across many edges |
Failure modes to raise before you are asked
- Stampede (thundering herd): a hot key expires and thousands of requests hit the database. Fix with request coalescing (single-flight), early probabilistic refresh, or stale-while-revalidate.
- Cold start: a new region or a flushed cache sends full load to the database. Warm it, or ramp traffic.
- Inconsistent invalidation: delete-then-write races. Prefer "write DB, then delete cache key" and short TTLs as a backstop.
- Hot keys: one celebrity key saturates one cache node. Replicate the key or add a small local in-process cache.
- Cache as the database: if the system cannot survive a cache flush, it is not a cache. Say whether yours can.
LLM-specific caching
Two different things share the name. Response caching returns a stored answer for an identical (or, with embeddings, similar) request; it is risky for personalised or time-sensitive answers. Prompt (prefix) caching is offered by model providers: a repeated prompt prefix such as a long system prompt or document is processed once and reused, cutting cost and time-to-first-token. Design prompts so the stable part comes first. Check your provider's current documentation for pricing and lifetime rules rather than quoting numbers from memory.
Practice
Written for this site to drill the lesson. Not reported interview questions.
- Your product page cache has a 5-minute TTL and the database falls over every 5 minutes at peak. Diagnose and fix.
- Design caching for an LLM support assistant where 30% of questions are near-duplicates, without leaking one customer's answer to another.
Check yourself
Name three fixes for a cache stampede.
Request coalescing (single-flight), early or probabilistic refresh, serving stale while revalidating.
What is the difference between response caching and prompt caching for LLMs?
Response caching reuses a whole answer; prompt caching reuses the provider's processing of a repeated prompt prefix.
Sources
- Official Anthropic docs: prompt caching