TechPrep
Production AI systemsReadLabPracticeCheck

Cost and latency

Model each request in tokens, then pull the levers in order: smaller model, shorter context, caching, batching, streaming.

Cost is roughly requests × (input tokens × input price + output tokens × output price). Latency is roughly time to first token + output tokens ÷ output speed, plus every tool call and retrieval round trip. Both are dominated by a few inputs you can measure.

Levers, roughly in the order to try them

  1. Route by difficulty: a small, fast model for easy requests, a large one for hard ones.
  2. Shrink context: retrieve fewer, better chunks; compact history; drop unused tool definitions.
  3. Cache: prompt prefix caching for stable instructions and documents; response caching where answers are not personal.
  4. Batch offline work through batch APIs where providers offer lower prices for asynchronous jobs.
  5. Stream so perceived latency is time to first token.
  6. Cap output: ask for concise formats and set max tokens.
  7. Parallelise independent calls; avoid sequential agent steps that could be one call.
Prices and discounts change often and differ by provider. Put current numbers into the calculator below from the provider's pricing page on the day; do not quote prices from memory in an interview, quote the model of the cost.

Token cost and latency calculator

Enter your own prices (per million tokens) from your provider's current pricing page.

Cost per request (no cache)-
Cost per request (with cache)-
Cost per day (with cache)-
Cost per 30 days (with cache)-
Output share of cost (no cache)-
Full response time (s)-

This ignores cache-write surcharges, batch discounts and tool-call round trips. Name them when you present an estimate.

Practice

Written for this site to drill the lesson. Not reported interview questions.

  1. Your agent averages 14 model calls per task. Propose three changes that halve cost without hurting the eval score, and how you would verify each.
  2. Leadership wants answers in under one second. Which parts of the latency budget can you control?

Check yourself

What dominates perceived latency in a streaming chat UI?

Time to first token.

Which lever usually comes first?

Routing easy requests to a smaller, cheaper model, measured against the eval.

Saved in this browser.