Path 7 of 10, by 2026 demand
AI Platform / ML Infrastructure Engineer
Build the platforms that serve, route, evaluate and observe models: inference, gateways, eval infrastructure, GPU and cluster operations.
Demand Infrastructure is the largest engineering group at OpenAI by title (51 titles name infrastructure; "Scaling" has 87 posts), and Anthropic lists inference, cluster and capacity roles (our count, §2.3). MLOps skills are described as table stakes for AI roles (third-party, M2).
What the role is
AI platform engineers build the shared systems that model-backed products run on: inference serving and autoscaling, LLM gateways, feature and embedding stores, evaluation pipelines, observability and cost attribution, and the Kubernetes and GPU fleet underneath.
At labs this extends to training and inference infrastructure at very large scale; at enterprises it is usually an internal AI platform for many product teams.
Level signals
- Operates stateful, high-throughput systems with clear SLOs.
- Capacity planning and cost efficiency (GPU utilisation, batching, caching).
- Platform product thinking: adoption, APIs other engineers love.
- Durability, consistency and failure-mode reasoning (official wording in Anthropic's agent-platform post, A5).
The typical loop
No single public source. Common shape for infrastructure roles.
| Stage | What it tests | Basis |
|---|---|---|
| Coding | Practical systems code: concurrency, parsing, data structures | Common industry pattern; no single source |
| Infrastructure system design | Serving, scheduling, autoscaling, multi-tenancy | Common industry pattern; no single source |
| Operations / debugging | Incident scenarios, Kubernetes and networking | Common industry pattern; no single source |
| Project deep dive | A platform you built or ran | Common industry pattern; no single source |
| Behavioural | Ownership, on-call, collaboration | Common industry pattern; no single source |
Track order
Work top to bottom. Ticks come from the lessons you mark complete.
- 01The trade-offsSystem design track. Reliability, queues and partitioning are the job.
- 02Worked promptsSystem design track. The LLM gateway prompt is a platform design.
- 03Production AI systemsLLM and agent engineering track. Agents, evals and cost from the platform side.
- 04The design methodSystem design track. Design method.
- 05Securing AI systemsSecurity engineering for AI systems track. Identity, tenancy and supply chain for model platforms.
- 06Arrays and stringsCoding patterns track. Coding fluency.
- 07Linked structures, trees and graphsCoding patterns track. Graphs for scheduling and dependency problems.
- 08Mock roundsMock rounds track. Design and deep-dive mocks.
Role practice set
Prompts written for this site. They are not reported interview questions.
- Design an inference service for an open-weights model on Kubernetes: autoscaling signal, batching, GPU bin-packing, cold starts, multi-tenant quotas.
- Design a nightly eval service that runs 50 teams' suites against new model versions and reports regressions.
- Debug scenario: p99 latency doubled after a model upgrade but GPU utilisation fell. Where do you look?
- Estimate GPUs needed for a given tokens-per-second target with an assumed per-GPU throughput you state.
Your résumé and story angle
- Kubernetes and Azure at scale are the core skills. [Cluster size, workloads, autoscaling or reliability work, numbers.]
- Platform as a product: [a platform other teams adopted; adoption numbers].
- AI integrations reframed as platform work: gateways, quotas, cost attribution. [Story.]
- Gap to close: GPU-specific operations and model serving. Build a small serving setup and be honest about depth.