Self-Attention, Step by Step

The one operation that powers every transformer: each token looks at every other token and pulls in what's relevant. It's just three matrices — Query, Key, Value — and the formula softmax(Q·Kᵀ / √d)·V.

query · key · valuedot-product scores softmax weightsweighted sumBYO-8 mini-GPT

Watch it compute

Tiny 2-D Q/K/V vectors keep the arithmetic legible. It opens on the first token's finished step — its Query scored against every Key, softmaxed, the Values blended. Press ▶ to replay, click a token to jump to its softmax, or type your own sentence.

① Scores = Q·Kᵀ, then ÷√d, then ② softmax → weights:

divide scores by √d (the “scaled” in scaled dot-product attention)

Full attention matrix (row = query, darker = more attention):

③ Output = Σ (weight × Value) — the context-mixed vector for the query token:

The problem attention solves

To understand a word you need its context. In "the cat ate the fish," the meaning of "ate" is tied to its subject ("cat") and object ("fish"), which sit several positions away. Older models pushed information along the sequence step by step (RNNs), which is slow and forgetful over long distances. Self-attention instead lets every token directly gather information from every other token in one parallel step — and learn which ones matter.

Query, Key, Value — the lookup metaphor

Attention reframes "understanding context" as a soft dictionary lookup. From each token's embedding, three small learned matrices produce three vectors:

Query (Q) — "what am I looking for?" The current token's question.
Key (K) — "what do I offer?" A label advertising what each token is about.
Value (V) — "what information do I pass on if attended to?" The actual content.

To process a token, you compare its Query against every token's Key with a dot product — high when they align, the way two arrows pointing the same way have a large dot product. Those scores become attention weights (via softmax), and the token's output is the weighted sum of all the Values. A token that matches your query contributes most of its Value; irrelevant tokens contribute almost nothing.

Why the details matter

Why the dot product? It measures alignment between a Query and a Key. Learning good Q/K projections is learning "which tokens should talk to which" — grammar, coreference, topic, all emerge here.
Why divide by √d? In high dimensions, dot products grow large, which makes softmax razor-sharp (one token gets weight ≈1) and kills the gradient. Scaling by √d keeps the scores tame so attention stays smooth and trainable. Toggle the checkbox to feel the difference.
Why softmax? It turns arbitrary scores into a clean probability distribution that sums to 1 — a set of mixing weights for the values.
"Self" attention means Q, K, V all come from the same sequence (tokens attend to each other). In cross-attention (e.g. decoder→encoder) the Queries come from one sequence and Keys/Values from another.

Multi-head attention just runs several of these in parallel with different learned projections, so one head can track syntax while another tracks coreference, then their outputs are concatenated. Stack that with a feed-forward network, add residual connections and normalization, and repeat — that's a transformer block.

The mask, and many heads

Two things the walkthrough above skipped, because a real transformer needs both. First: not every token may look at every other token. A decoder predicting token 4 must never see tokens 5 and 6 — that would let training copy the answer instead of predicting it — so a causal mask sets those scores to −∞ before the softmax, and a padding mask does the same for the filler tokens a batch adds so every sequence in it has the same length. Second: real models don't run one wide attention — they split the same total dimension into several narrower heads that each learn a different kind of relation, then concatenate the results back together. Pick a mask below, then compare 1 head against 4 on the six-token sentence "the cat sat on the mat":

mask

heads: 1

Check yourself

Takeaways: attention is a soft, learnable lookup. Each token emits a Query, a Key, and a Value; score every Query·Key, scale by √d, softmax into weights, and output the weighted sum of Values. It lets every token gather context from every other token in parallel — the core of every modern LLM. Build the whole block in BYO-8 (mini-GPT). For a real running model, poke the Transformer Explainer.