Production AI systems
Agent orchestration, evals, guardrails, and cost and latency: what separates a demo from a system.
Lessons
- 01Agent orchestration: workflows before agentsUse the simplest composition that works: a single call, a fixed workflow, and only then an autonomous agent loop with limits.
- 02Evals: how you know it worksA versioned test set, metrics that match the task, graders you have checked, and a gate on every prompt or model change.
- 03GuardrailsLayered checks on inputs, outputs and actions, chosen from the harms that matter for this product rather than a generic list.
- 04Cost and latencyModel each request in tokens, then pull the levers in order: smaller model, shorter context, caching, batching, streaming.