TechPrep
Worked promptsReadPracticeCheck

Worked prompt: a rate limiter for a public API

A Senior-level classic, answered with Staff-level trade-offs: algorithm choice, distributed counting and failure behaviour.

This prompt was written for this site to practise the method. It is not a reported interview question from any company.

Prompt. Design rate limiting for a public API used by 50,000 customers. Limits are per API key and per endpoint, configurable per plan.

1. Requirements

Functional: per-key, per-endpoint limits; return 429 with Retry-After; plan-based config; an admin override.

Non-functional: adds under ~1 ms p99 at the gateway; correct within a few percent; never the reason the API is down.

2. Estimates

Assume 200k requests/s at peak across all keys. One counter update per request: a single Redis primary can handle this order of load, but one hot key could not sit on one shard forever. Partition counters by key.

3. Algorithm options

Fixed window: simplest, but allows 2x bursts at window edges.

Sliding window log: exact, but stores every timestamp; memory grows with the limit.

Sliding window counter: weights the previous window; close to exact with two counters. Good default.

Token bucket: allows controlled bursts, natural for "100/min with bursts of 20". Choose this if the product wants bursts.

Decision: token bucket for customer-facing limits (matches how plans are sold), implemented atomically in Redis with a Lua script or equivalent.

4. Architecture

Limiter runs in the API gateway layer (or as a sidecar on Kubernetes). Config is cached locally and refreshed by push. Counters live in a sharded Redis cluster keyed by key:endpoint.

For very high-volume keys, use local token buckets per gateway node that periodically sync with the central store: accuracy drops slightly, latency and load drop a lot.

5. Failure behaviour (the Staff part)

If Redis is unavailable: fail open with local per-node limits as a backstop, and alert. Failing closed turns a cache outage into a full API outage.

Clock skew is irrelevant if the store's time or monotonic counters are used.

Abuse versus fairness: separate DDoS protection (edge/WAF) from commercial rate limits; they have different owners and SLOs.

6. Wrap-up

Measure: limiter latency, rejection rate by plan, false rejections reported by customers. Next: per-tenant cost-based limits (weight requests by expense), which matters for LLM endpoints where one request can cost 1,000x another.

Practice

Written for this site to drill the lesson. Not reported interview questions.

  1. Change one requirement: limits must be exact across regions. What changes and what does it cost?
  2. Adapt the design to limit an LLM API by tokens per minute rather than requests per minute.

Check yourself

Why fail open?

So an outage of the limiter's store does not become an outage of the API; local limits remain as a backstop.

Which algorithm fits plans sold as a rate with bursts?

Token bucket.

Saved in this browser.