How LLM features fail
A working mental model of tokens, context and sampling, organised around the ways production features break.
You do not need to derive attention to pass an AI Engineer loop. You do need to predict how an LLM feature will misbehave and design around it. Most failures fall into five groups.
| Failure | What causes it | First line of defence |
|---|---|---|
| Confident wrong answers (hallucination) | The model generates plausible text without grounding | Retrieval with citations; instruct it to say "not in the sources"; check faithfulness in evals |
| Ignored instructions | Long or conflicting context; instruction buried mid-prompt | Clear structure, important rules at the start or end, fewer competing instructions |
| Format breaks | Free-text generation where a parser expects JSON | Structured outputs / tool schemas; validate and retry |
| Inconsistency | Sampling and prompt sensitivity | Lower temperature for deterministic tasks; evals over many samples, not one |
| Cost and latency blow-ups | Context grows with history and retrieved text; agents loop | Budgets per request, summarisation/compaction, caching, step limits |
Words to use precisely
- Token: the unit models read and are billed in; roughly three-quarters of an English word, but it varies by tokenizer and language.
- Context window: the maximum tokens of input plus output per call. Bigger is not free: cost and latency scale with input, and quality can degrade on long inputs.
- Time to first token (TTFT) versus output tokens per second: users feel the first; long answers feel the second.
- Temperature: randomness of sampling. Not a quality knob.
Interview signal: when asked "how would you build X with an LLM", start with how you will know it works (the eval), then build. Both labs' Applied AI posts put evaluation right next to building, and OpenAI's says success is "production systems ... not simply ... successful demonstrations" (Sources A6, O7).
Practice
Written for this site to drill the lesson. Not reported interview questions.
- A summarisation feature works in demos but users report it invents action items. Walk through diagnosis and fixes.
- Explain to a product manager why doubling the context window doubled the bill but not the quality.
Check yourself
What should you design first for an LLM feature?
How you will measure that it works: the eval.
Which latency metric do chat users feel most?
Time to first token.
Saved in this browser.