Chinchilla: compute-optimal scaling

Given a fixed compute budget, should you train a bigger model or feed a smaller one more tokens? Chinchilla's answer upended the field: the giants of 2020–21 were badly undertrained — for the same compute, a model a quarter of Gopher's size, fed 4× the data, beats it. The rule of thumb that fell out (≈20 tokens per parameter) still shapes how every serious LLM budget is spent.

compute budgetparams vs tokensthe 20:1 ruleundertrained giantsscaling laws

📄 The paper: Training Compute-Optimal Large Language Models — Hoffmann et al. (DeepMind) (2022) · read it ↗. Scenes below are generated from a storyboard spec ↗ — read the paper with the three-pass method.

The bet the field had made — and the paper's counter

Before Chinchilla, scaling wisdom said: with more compute, grow the model — data can lag. Gopher was the poster child: 280B parameters trained on ~300B tokens. Chinchilla spent the same training compute the other way: a 70B-parameter model fed 1.4T tokens — 4× smaller, 4.7× more data. It won on almost every benchmark. The two towers below are the whole argument: same budget, opposite shapes.

The loss landscape over params × tokens

The paper fit loss as a function of model size N and training tokens D across 400+ runs. Picture the result as a surface: quality improves toward the top-right (more of both — more compute), but for any FIXED budget you live on one anti-diagonal of this map, and the best point on that diagonal is the balanced one. Straying to either corner — a giant model starved of data, or a tiny model drowning in it — wastes the same compute on a worse loss. Click the chips to see where the three regimes land.

Scrub the budget split. Same compute at every step — only the model-size / data trade changes.

The rule of thumb: ≈20 tokens per parameter

Fitting the frontier gives the memorable prescription: for a compute-optimal run, train on roughly 20 tokens for every parameter. A 70B model wants ~1.4T tokens — exactly Chinchilla. The rule reshaped budgets overnight, and its descendants matter too: models like Llama are deliberately trained past compute-optimal (more tokens than 20:1), trading extra training compute for a smaller, cheaper-to-serve model — a different optimum once inference cost enters the ledger (your Phase 9 cost-model instincts, applied to training).

Two beats: the line, then how to read your own model size off it.

The trap the paper corrects: "more parameters = better model" — for a fixed budget it's false. The pre-Chinchilla giants sat in the red corner of the surface above: enormous N, starved D. When someone proposes scaling the model, the compute-optimal question is "scaled against how many tokens?" — and if serving cost matters (it always does), the answer shifts token-heavy past 20:1.

Takeaways: For a fixed training budget, loss is minimised by growing parameters and tokens together — roughly 20 tokens per parameter at the compute-optimal point. The 2021 giants were undertrained; Chinchilla (70B × 1.4T) beat Gopher (280B × 0.3T) on the same compute. And the modern twist: deliberately overtraining smaller models (Llama-style) trades training compute for cheaper inference — pick your point on the frontier with the whole cost model in view.

Companions: the paper (arXiv) · 🎨 LLM internals walkthrough · 🧭 Phase 9 · cost models · DeepMind's summary