In Java, "the third argument to this method" is a fact about a call site, unrelated to any other call. Self-attention has no such fact built in at all: positional encoding stamped a fingerprint onto each token so the model could tell positions apart, but a model that generates one token at a time needs something stronger — a position scheme that survives being applied one row at a time, and a way to stop re-doing yesterday's arithmetic every time it writes a new word. Those are two different problems, and this page builds both: a rotation that encodes relative position for free, and a cache that turns generation from quadratic into linear.
Sinusoidal positional encoding adds a fixed vector to each
token's embedding — one fingerprint per absolute position. That's enough to break the symmetry that makes
attention order-blind, but it hands the model an absolute coordinate and leaves it to figure out
"these two tokens are 5 apart" by comparing two fingerprints indirectly. Rotary position embedding
(RoPE) takes a different approach: instead of adding a position vector to the embedding, it
rotates the Query and Key vectors themselves, by an angle proportional to position. Rotating both by
the same amount before taking their dot product is exactly equivalent to rotating one of them by the
difference in their positions — so the relationship a query at position m has with a
key at position n depends only on m − n, never on m and
n individually. That's the fact Inside an LLM
builds on ("relative position, encoded structurally") without ever drawing why it's true. Drag the position
below and watch it happen — pick a base, then watch how far apart in tokens a dimension can count before
its rotation wraps back to where it started:
Radians turned per token, one bar per dimension-pair (dim 0 → dim 62, this base):
Rotation fixes where a token's Query and Key point; it says nothing about how much work generating the next token costs. Naively, producing token 64 of a reply means re-running the whole 1-through-64 forward pass — recomputing Keys and Values for 63 tokens whose embeddings never changed. The KV cache stores every past token's Key and Value the first time they're computed and never touches them again: step 64 projects one new token, appends it to two growing stacks, and attends the new Query against everything already sitting in the cache.
m − n directly and never changes a
vector's length; different dimension-pairs rotate at different speeds, and a bigger base
slows the slowest pairs down further, buying more context before they wrap. The KV cache stores every past
token's Key and Value once and never recomputes them, turning per-token generation cost from growing with
the transcript into a constant.Curated companions: EleutherAI — Rotary Embeddings · Kipply — Transformer Inference Arithmetic (KV-cache memory).