Worked prompt: a multi-tenant LLM gateway
A Staff-level, AI-era design: routing, quotas, streaming, cost attribution, safety hooks and provider failure in one service.
Prompt. Your company has 40 internal teams calling several model providers directly with shared keys. Design a gateway every LLM call goes through.
1. Why this exists (frame it)
Problems today: no cost attribution, leaked keys, no central policy for PII or logging, outages when one provider rate-limits, no evals on model changes. The gateway is a platform product; its users are engineers. Say who owns it and how teams adopt it.
2. Requirements
Functional: OpenAI- and Anthropic-style APIs (or one normalised API), streaming, tool calls, per-team keys and quotas, model routing and fallback, request/response logging with redaction, cost reporting.
Non-functional: adds under ~20 ms to time-to-first-token; survives a provider outage; never stores raw secrets or unredacted PII in logs; per-team isolation.
3. Estimates
Assume 300 requests/s at peak, average 4k input and 500 output tokens, streams lasting up to 60 s. Concurrency is the constraint, not QPS: 300/s × ~20 s average ≈ 6,000 concurrent streams. That rules out thread-per-request designs and points to async I/O.
4. Architecture
Stateless gateway pods behind a load balancer (Kubernetes, scaled on concurrent streams, not CPU).
Control plane: team registry, keys (stored as hashes; provider keys in a vault), quotas, routing policy, model allowlist.
Data plane per request: authenticate, check quota (token-bucket by estimated tokens, reconciled with actual usage after the response), apply input policy (PII redaction, prompt-injection heuristics where relevant), route, stream back, apply output policy, emit a usage event.
Usage events to a stream, then to a cost warehouse for chargeback and dashboards.
5. Deep dive: routing and failure
Routing policy per team: preferred model, allowed fallbacks, max cost per request.
On provider 429s or 5xxs: retry with backoff once within the caller's deadline, then fall back to an approved model. Fallbacks change behaviour, so teams opt in, and responses carry a header saying which model answered.
Circuit breaker per provider and region; health is measured on time-to-first-token as well as errors.
6. Deep dive: evals and change control
A model or prompt change is a deploy. The gateway records which model and version served each request, so teams can run their eval suites against a new model before switching, and roll back by config.
7. Trade-offs to state
Normalised API (easy switching, lowest common denominator) versus pass-through (full features, more lock-in). A common answer: pass-through with a thin shared envelope for auth, quotas and logging.
Logging full prompts helps debugging and evals but raises data-protection risk; default to redacted, time-limited retention with per-team opt-in.
Practice
Written for this site to drill the lesson. Not reported interview questions.
- Add a requirement: some teams must use only models hosted in your own cloud tenancy. What changes?
- How would you detect that one team's agent loop is burning 30% of the monthly budget, and what should happen automatically?
- Present the gateway to 40 team leads in five minutes. What do you lead with?
Check yourself
What sizes an LLM gateway: QPS or concurrency?
Concurrency: long-lived streams multiply requests per second by stream duration.
Why must fallbacks be opt-in?
A different model can behave differently; teams need to have evaluated it.