TechPrep
The trade-offsReadPracticeCheck

Caching and its failure modes

Caches are the cheapest latency win and the most common source of subtle bugs: plan invalidation, stampedes and cold starts up front.

PatternHow it worksGood forRisk
Cache-asideApp reads cache, on miss reads DB and fills cacheGeneral read-heavy dataStale data until TTL or explicit invalidation
Write-throughWrites go to cache and DB togetherRead-after-write consistencyWrite latency; cache full of cold data
Write-behindWrite to cache, flush to DB asynchronouslyBursty writesData loss if the cache dies before flushing
Read-through / CDNCache sits in front and fetches on missStatic and semi-static contentInvalidation across many edges

Failure modes to raise before you are asked

LLM-specific caching

Two different things share the name. Response caching returns a stored answer for an identical (or, with embeddings, similar) request; it is risky for personalised or time-sensitive answers. Prompt (prefix) caching is offered by model providers: a repeated prompt prefix such as a long system prompt or document is processed once and reused, cutting cost and time-to-first-token. Design prompts so the stable part comes first. Check your provider's current documentation for pricing and lifetime rules rather than quoting numbers from memory.

Practice

Written for this site to drill the lesson. Not reported interview questions.

  1. Your product page cache has a 5-minute TTL and the database falls over every 5 minutes at peak. Diagnose and fix.
  2. Design caching for an LLM support assistant where 30% of questions are near-duplicates, without leaking one customer's answer to another.

Check yourself

Name three fixes for a cache stampede.

Request coalescing (single-flight), early or probabilistic refresh, serving stale while revalidating.

What is the difference between response caching and prompt caching for LLMs?

Response caching reuses a whole answer; prompt caching reuses the provider's processing of a repeated prompt prefix.

Sources

Saved in this browser.