Attention Is All You Need: replacing recurrence with self-attention

Before 2017, a sequence model read left to right: an RNN's state at position n depended on its state at n-1, so two distant words could only talk to each other through n sequential steps — and nothing could compute in parallel across time. This paper's bet: drop recurrence and convolution entirely. Stack layers of self-attention instead, and every pair of positions is one matrix multiply apart — a constant path length, computed for the whole sequence in parallel. The title undersold it.

self-attentionsoftmax(QKᵀ/√d_k)Vmulti-headpositional encodingconstant path length

📄 The paper: Attention Is All You Need — Vaswani et al. (Google Brain / Google Research) (2017) · read it ↗. Scenes below are generated from a storyboard spec ↗ — read the paper with the three-pass method.

Scaled dot-product attention — and why the ÷√d_k is load-bearing

Every position emits a Query, and every position (itself included) offers a Key and a Value. Score query against every key with a dot product, turn the scores into weights with softmax, then mix the Values by those weights: softmax(Q·Kᵀ / √d_k)·V. The paper's Section 3.2.1 flags a specific failure mode: for large d_k, raw dot products grow large in magnitude, which pushes softmax toward one-hot — and a near one-hot softmax has near-zero gradient everywhere except its winning cell. Dividing by √d_k is the fix. Below is a real run: a 4-token toy sequence, hand-picked d_k=8 query/key vectors (not learned — chosen to make the effect visible), scored both ways.

Grid colour is the softmax weight itself: white ≈ 0, saturated green ≈ 1. Rows are queries, columns are keys.

Multi-head attention: eight cheap views instead of one expensive one

One attention computation learns one notion of relevance. The paper instead projects Q, K, V into h=8 smaller subspaces (d_k=d_v=d_model/h=64), runs attention in each independently, then concatenates and projects back. The parameter count is the striking part: a single full-width head needs 4 projections of 512×512 — 1,048,576 parameters. Eight heads need 8 × 3 projections of 512×64 plus one output projection of 512×512 — 786,432 + 262,144 = 1,048,576. Identical. Section 3.2.2 makes the same point: splitting into heads is not an extra cost, it is a different shape for the same budget, and each head is free to specialise (the paper's appendix shows heads that track syntax, others that track coreference).

Blue = one full-width head — 4×512² = 1,048,576 params. Green = the same budget cut into 8 slivers — 8×(3×512×64)+512² = 1,048,576 params. Same total either way.

Positional encoding: self-attention has no notion of order until you add one

Attention is a weighted sum over a set of positions — shuffle the input tokens and, without help, the weighted sums shuffle identically. The paper injects order with a fixed sinusoid added to each token's embedding: PE(pos,2i)=sin(pos/10000^(2i/d_model)), PE(pos,2i+1)=cos(pos/10000^(2i/d_model)). Each dimension-pair i oscillates at its own wavelength, geometrically spaced from 2π (the fastest pair) to 10000·2π (the slowest) — Section 3.5's own description. Below is a real run for a toy d_model=16, 16 positions.

Grid colour is the raw PE value in [-1,1]: blue = negative, white = 0, green = positive — a real run of the formula above, no rounding beyond display.

The trap the paper corrects: "Self-attention has no notion of position, so it must be worse at handling order than an RNN." Backwards: self-attention has no order until you add one (positional encoding) — but once you do, every layer connects any two positions in a single hop, where an RNN needs to loop through every position between them. And multi-head attention isn't more expensive than single-head: splitting into 8 heads costs the identical parameter count, not 8× — it buys diversity of views for free, not extra capacity.

Takeaways: Scaling by √d_k is a mechanical fix for softmax saturation: in this toy run it roughly doubles the attention distribution's entropy (0.65 → 1.65 bits) rather than concentrating weight on one key. Splitting into h=8 heads costs exactly as many parameters as one full-width head — 1,048,576 either way — so multi-head is extra views at no extra parameter budget. And because self-attention has no built-in order, positional encoding injects one via sinusoids whose wavelengths span 2π to 10000·2π, giving the model a fixed coordinate system it can read relative offsets from.

Companions: the paper (arXiv) · 🎨 self-attention, hands-on QKV · 🎨 positional encoding, hands-on · 🧭 Phase 5 · NLP & Transformers