Design ChatGPT

In a Java service you scale by adding stateless replicas until QPS fits. An LLM replica is different: its throughput is set by how fast it can stream its weights and its KV cache out of GPU memory, so the number to derive is tokens per second per replica — and that depends on batch size, which depends on KV memory per request.

45 min round 9 computed numbers 4-part eval plan 8-point rubric

⚠️ Planning numbers, not measurements. Hardware and model facts are dated on the numbers sheet; traffic, prices and efficiency are labelled assumptions. Replace them with your own measurements (the vLLM load-test exercise) before quoting them.

The prompt

Design a ChatGPT-style chat service for 10 million daily users. State your numbers.

Attempt it first: the 45-minute round

Set the timer, answer out loud or on paper, then score yourself against the rubric before you read the model answer below.

Framing (5 min) — Ask questions, fix the scale and the latency/quality/cost targets, state assumptions with numbers.
Architecture (10 min) — Draw the boxes end to end: data in, the model call, storage, serving, feedback.
Deep dive (15 min) — Pick the hardest part and do the arithmetic: tokens/s, KV memory, QPS → replicas, cost per 1,000 requests.
Trade-offs (10 min) — Name what you would trade (batch vs latency, quality vs cost, build vs buy) and what breaks.
Wrap-up (5 min) — Evaluation plan, monitoring, failure modes, and what you would do next.

45:00

Not started

Self-grade

Asked clarifying questions and stated the scale before drawing
Every number came with its assumption and units
Named the binding constraint (memory, latency or cost) and showed the arithmetic
Gave at least one trade-off with what it costs
Said how you would evaluate it and what you would monitor
Derived peak QPS from users, not a round number
Computed KV memory per request and used it to bound batch
Converted tokens/s into replicas and GPUs, and stated prefill was ignored

Tick what you did. 0 of 8.

The model answer, step by step

Five steps, one tap each. The readout gives the step's answer and lists the numbers it uses; the same numbers are highlighted in the table underneath.

Design ChatGPT, one tap per step

👉 Predict first, then tap. Every number below is computed from the assumptions in the table.

1. Framing

5 min

2. Architecture

10 min

3. Deep dive

15 min

4. Trade-offs

10 min

5. Wrap-up

5 min

Tap a step above.

Quantity	How it is computed	Value
Daily active users	assumption	10,000,000
Requests per user per day	assumption	8
Share of daily traffic in the peak hour	assumption	15%
Input tokens per request (history + prompt)	assumption	1,500
Output tokens per request	assumption	400
GPUs per replica (tensor parallel), 70B at 1 byte per weight	assumption	4
Concurrent sequences per replica (batch)	assumption	128
Peak requests per second	dau × rpd × peak ÷ 3600	3,333
KV cache per request (fp16, 70B, in+out tokens)	2 × 80 layers × 8 kv-heads × 128 × seq × 2 bytes	623 MB
Most sequences that fit (90% of 4×80 GB, minus 70 GB weights)	⌊(0.9 × 4 × 80 GB − 70 GB) ÷ KV per request⌋	350
One decode step at that batch	(70 GB + batch × KV) ÷ 4 GPUs ÷ 3.35 TB/s	11.17 ms
Output tokens per second per replica	batch ÷ step	11,458 tok/s
Output tokens per second needed at peak	qps × output tokens	1,333,333 tok/s
Replicas	⌈needed ÷ per-replica⌉	117
GPUs	replicas × 4	468
Cost per 1,000 requests at $4/GPU-hour	replica $/hour ÷ (requests per hour per replica) × 1000	$0.155

Takeaway. The headline figure — replicas — is 117 (⌈needed ÷ per-replica⌉). Say the assumption, show the formula, then give the number.

The evaluation plan

A 500-prompt golden set replayed on every model or prompt change, scored by rubric-based LLM judge checked against 100 human labels.
Online: thumbs-down rate, regenerate rate and conversation length, with a holdback group.
Safety regression set (jailbreaks, self-harm, PII) that must not get worse.
Latency SLOs as tests: p95 first token under 800 ms in the load-test.

Go deeper on this site

ML & GenAI system design — the framework and the numbers, interactively.
Interview practice — the wider interview round.
The numbers sheet — formulas and dated sources used on all twelve designs.

Design ChatGPT

The prompt

Attempt it first: the 45-minute round

Self-grade

The model answer, step by step

Design ChatGPT, one tap per step

1. Framing

2. Architecture

3. Deep dive

4. Trade-offs

5. Wrap-up

The evaluation plan

Go deeper on this site

Check yourself