A Transformer sees all tokens at once and has no built-in notion of order — "dog bites man" and "man bites dog" would look identical. Positional encoding fixes that by stamping each position with a unique pattern of sine and cosine waves, added straight into the embeddings.
Self-attention computes how much every token attends to every other, but the operation is permutation-invariant: shuffle the inputs and you get the same set of outputs, just reordered. There's nowhere for "this word came third" to live. Recurrent nets got order for free by reading left-to-right; a Transformer reads everything in parallel, so order has to be injected. The fix is to add, to each token's embedding, a vector that depends only on its position.
The trick: build that position vector from sinusoids of many different frequencies. Low dimensions use fast waves (they flip between adjacent tokens); high dimensions use slow waves (they change over long spans of text). Each position lands on a unique combination — a fingerprint — and because the waves are smooth, nearby positions get similar encodings, which lets the model reason about how far apart two tokens are. The heatmap below is the encoding for every position (rows) across every dimension (columns); pick a position to see its vector:
The selected position's encoding vector (one value per dimension):
Reading a single column down the heatmap traces one sinusoid across positions. Low-index columns oscillate quickly; high-index columns oscillate slowly — exactly like the second, minute, and hour hands of a clock together pinpointing a time. Two example columns:
Because each frequency component shifts predictably with position, the encoding for position
p + k is a fixed linear function of the encoding for p — so the model can learn to
attend "three tokens back" regardless of where it is in the sequence. The vectors are simply added to
the token embeddings (same dimension), so a token's final representation carries both what it is and
where it is. (Modern models often use learned or rotary variants, but the sinusoidal idea is the
foundation.)
Curated companions: Jay Alammar — The Illustrated Transformer (positional-encoding section) · Transformer Explainer.