TechPrep
Worked promptsReadPracticeCheck

Worked prompt: a multi-tenant LLM gateway

A Staff-level, AI-era design: routing, quotas, streaming, cost attribution, safety hooks and provider failure in one service.

This prompt was written for this site to practise the method. It is not a reported interview question from any company.

Prompt. Your company has 40 internal teams calling several model providers directly with shared keys. Design a gateway every LLM call goes through.

1. Why this exists (frame it)

Problems today: no cost attribution, leaked keys, no central policy for PII or logging, outages when one provider rate-limits, no evals on model changes. The gateway is a platform product; its users are engineers. Say who owns it and how teams adopt it.

2. Requirements

Functional: OpenAI- and Anthropic-style APIs (or one normalised API), streaming, tool calls, per-team keys and quotas, model routing and fallback, request/response logging with redaction, cost reporting.

Non-functional: adds under ~20 ms to time-to-first-token; survives a provider outage; never stores raw secrets or unredacted PII in logs; per-team isolation.

3. Estimates

Assume 300 requests/s at peak, average 4k input and 500 output tokens, streams lasting up to 60 s. Concurrency is the constraint, not QPS: 300/s × ~20 s average ≈ 6,000 concurrent streams. That rules out thread-per-request designs and points to async I/O.

4. Architecture

Stateless gateway pods behind a load balancer (Kubernetes, scaled on concurrent streams, not CPU).

Control plane: team registry, keys (stored as hashes; provider keys in a vault), quotas, routing policy, model allowlist.

Data plane per request: authenticate, check quota (token-bucket by estimated tokens, reconciled with actual usage after the response), apply input policy (PII redaction, prompt-injection heuristics where relevant), route, stream back, apply output policy, emit a usage event.

Usage events to a stream, then to a cost warehouse for chargeback and dashboards.

5. Deep dive: routing and failure

Routing policy per team: preferred model, allowed fallbacks, max cost per request.

On provider 429s or 5xxs: retry with backoff once within the caller's deadline, then fall back to an approved model. Fallbacks change behaviour, so teams opt in, and responses carry a header saying which model answered.

Circuit breaker per provider and region; health is measured on time-to-first-token as well as errors.

6. Deep dive: evals and change control

A model or prompt change is a deploy. The gateway records which model and version served each request, so teams can run their eval suites against a new model before switching, and roll back by config.

7. Trade-offs to state

Normalised API (easy switching, lowest common denominator) versus pass-through (full features, more lock-in). A common answer: pass-through with a thin shared envelope for auth, quotas and logging.

Logging full prompts helps debugging and evals but raises data-protection risk; default to redacted, time-limited retention with per-team opt-in.

Practice

Written for this site to drill the lesson. Not reported interview questions.

  1. Add a requirement: some teams must use only models hosted in your own cloud tenancy. What changes?
  2. How would you detect that one team's agent loop is burning 30% of the monthly budget, and what should happen automatically?
  3. Present the gateway to 40 team leads in five minutes. What do you lead with?

Check yourself

What sizes an LLM gateway: QPS or concurrency?

Concurrency: long-lived streams multiply requests per second by stream duration.

Why must fallbacks be opt-in?

A different model can behave differently; teams need to have evaluated it.

Saved in this browser.