The Transformer, assembled

You've already driven the parts: self-attention computing Q·Kᵀ→softmax→·V live, and positional encoding stamping order onto order-blind math. This page assembles them into the 2017 architecture that ended the RNN era and powers every modern LLM — and lets you poke the three ideas that made it work: many heads, residual highways, and total parallelism (paid for in n²).

multi-head attentionresiduals + LayerNorm encoder vs decodern² attention

Why recurrence lost

An LSTM reads a sentence like a human with amnesia: word by word, cramming everything so far into one fixed-size memory. Two fatal costs: long-range information decays (even gated), and — worse for the GPU age — step 47 can't start before step 46 finishes. The Transformer's move: drop recurrence entirely. Let every token directly look at every other token (attention), all at once, in one giant matrix multiply. Order, which recurrence got for free, is bolted back on with positional encodings. The result trains on the whole sequence in parallel — which is the real reason it, and not the LSTM, could be scaled to the entire internet.

The block, clickable

A GPT-style decoder stack — click any stage

Many heads see many things

One attention pattern per layer is not enough: "who does it refer to?" and "what's the verb's object?" are different questions about the same sentence. So each layer runs several heads — independent smaller Q/K/V projections — in parallel, and concatenates their answers. Click a token below and read three heads' attention (patterns are hand-built illustrations of the kinds real heads learn — probe a live GPT-2 in the poloclub companion): one tracks the previous word (syntax glue), one resolves references, one binds verbs to objects. Then flip the causal mask: a GPT decoder must not peek at future tokens during generation, so everything after the clicked token is forced to zero and the remaining weights renormalize — that mask is the entire difference between reading (BERT) and writing (GPT).

Multi-head attention on one sentence

causal mask (decoder mode — no peeking at the future)

Residual highways

A GPT stacks 12–96 of those blocks. Phase 2 taught you what happens to a gradient multiplied through that many layers: it vanishes (or explodes). The transformer's fix is the same one ResNet used in Phase 4: every sub-layer's output is added to its input — x + Attention(x), then x + FFN(x) — so there is always an untouched identity path from the loss all the way down to layer 1. LayerNorm keeps each token's activations in a healthy range so the additions never blow up. Drag the slider: it sets how much each block's transformation shrinks the gradient, and the bars show what survives at depth — with and without the highway.

What reaches layer 1 of a 12-block stack?

per-block gradient factor how much of the gradient each block lets through — the chain rule multiplies these

Without residuals the signal is a pure product — at 0.7× per block, 0.7¹² ≈ 0.014 reaches layer 1: it learns nothing. With residuals, backprop always includes the pure-identity path — a gradient term of strength 1 arrives at every depth, whatever the blocks do (and LayerNorm keeps the added parts from exploding). Same lesson as your Phase 2 vanishing-gradients drill — now you know why 96-layer models train.

What all this costs

Two numbers dominate transformer economics. Parameters grow ≈ 12·L·d² (each of L blocks holds attention + FFN matrices of size ~d²) plus the embedding table. Attention compute/memory grows with the square of context length — every token attends to every token, an n×n matrix per head per layer. Double the context, pay 4×. That n² is why long-context models are expensive, why chunking exists in Phase 6's RAG, and why a small army of "efficient attention" papers exists. Slide and watch:

Size & cost calculator

layers L depth: how many attention+FFN blocks are stacked

width d_model width: the size of every token's vector — parameters grow with its SQUARE

context length n how many tokens fit in one pass — attention cost grows with its square

The tower, to scale: each slab is one transformer block (attention + FFN). Drag layers L and watch depth grow; d_model widens every slab. This is the shape the parameter count above is really measuring.

Encoder, decoder, and which one ate the world

The 2017 paper had two towers: an encoder (sees the whole input, no mask — great at understanding: classification, NER, retrieval embeddings → BERT's family) and a decoder (causal mask, generates left-to-right → GPT's family). Modern LLMs are almost all decoder-only: one tower, one objective — predict the next token — scaled until understanding emerged as a side effect. You'll build exactly that in BYO-8 (mini-GPT), and Phase 6B dissects what the industrial versions add (BPE, MoE, LoRA/quantization).

⚠️ Traps: attention itself has no order — positional encoding is load-bearing, not decoration · residuals are for gradient flow in deep stacks, not accuracy garnish · multi-head ≠ more attention, it's differently-projected attention in parallel · the causal mask is applied before softmax (set to −∞), so future weights are exactly 0, not merely small · n² is in sequence length, not parameters — a small model on a huge context can cost more than a big model on a short one.

Takeaways: transformer = attention (talk) + FFN (think), wrapped in residual + LayerNorm, stacked deep · heads = parallel questions · mask = the read/write switch (BERT vs GPT) · parallel training bought the scale that changed everything; n² is the bill · you have now met every part — go build one: BYO-8 mini-GPT, with Karpathy as your co-pilot. Next phase: LLMs put to work — RAG.

Curated companions: Jay Alammar — The Illustrated Transformer · poloclub — Transformer Explainer (a live GPT-2).