Cost and latency
Model each request in tokens, then pull the levers in order: smaller model, shorter context, caching, batching, streaming.
Cost is roughly requests × (input tokens × input price + output tokens × output price). Latency is roughly time to first token + output tokens ÷ output speed, plus every tool call and retrieval round trip. Both are dominated by a few inputs you can measure.
Levers, roughly in the order to try them
- Route by difficulty: a small, fast model for easy requests, a large one for hard ones.
- Shrink context: retrieve fewer, better chunks; compact history; drop unused tool definitions.
- Cache: prompt prefix caching for stable instructions and documents; response caching where answers are not personal.
- Batch offline work through batch APIs where providers offer lower prices for asynchronous jobs.
- Stream so perceived latency is time to first token.
- Cap output: ask for concise formats and set max tokens.
- Parallelise independent calls; avoid sequential agent steps that could be one call.
Token cost and latency calculator
Enter your own prices (per million tokens) from your provider's current pricing page.
This ignores cache-write surcharges, batch discounts and tool-call round trips. Name them when you present an estimate.
Practice
Written for this site to drill the lesson. Not reported interview questions.
- Your agent averages 14 model calls per task. Propose three changes that halve cost without hurting the eval score, and how you would verify each.
- Leadership wants answers in under one second. Which parts of the latency budget can you control?
Check yourself
What dominates perceived latency in a streaming chat UI?
Time to first token.
Which lever usually comes first?
Routing easy requests to a smaller, cheaper model, measured against the eval.