Embeddings & Cosine Similarity

What is an embedding?

Computers don't understand words — they understand numbers. An embedding is how we bridge that gap: it's a vector (a list of numbers) that represents a piece of meaning — a word, a sentence, an image, even a user. The magic isn't the numbers themselves; it's how they're arranged. Embedding models are trained so that things with similar meaning land close together in the vector space, and unrelated things land far apart.

That one property is enormously powerful, because it turns meaning into geometry. Once meaning is geometry, three hard problems become easy:

• "How similar are these?" → measure the angle between their vectors (Part 1).
• "Find me things like this" → nearest-neighbor search in the space (Part 2).
• "king is to man as ? is to woman" → do algebra on the vectors (Part 3).

Real embeddings live in hundreds or thousands of dimensions (you can't draw those), but every intuition below works the same in 2-D, so we'll stay where you can see it.

1 · Cosine similarity = direction, not length

To use embeddings we need to measure similarity. The standard tool is cosine similarity: the cosine of the angle between two vectors, (A·B)/(|A||B|). Why the angle and not plain distance? Because in embedding spaces, direction carries the meaning and length often just reflects incidental things like text length or word frequency. Cosine deliberately ignores length — a short query and a long document can score a perfect 1.0 if they point the same way. It ranges from 1 (same direction → very similar), through 0 (perpendicular → unrelated), to −1 (opposite → contrasting).

Drag the tips of A and B and watch the number move. Then tick "show unit vectors": both arrows snap to length 1 but the cosine doesn't change — proof that only the angle matters.

2 · Search by meaning

Old-style keyword search matches exact words: search "car" and you miss a document that only says "automobile." Semantic search fixes this by comparing embeddings instead of words — so text with the same meaning scores high even when it shares no vocabulary. Below, each sentence is embedded and ranked by cosine similarity to your chosen query. Watch the pairs that mean the same thing ("the cat sat on the warm mat" / "a kitten rested on a cozy rug") rise to the top.

(Our toy uses hand-picked 8-dimensional feature vectors — not word counts — so "cat"↔"kitten" and "fell"↔"dropped" already score high with zero shared words; a real embedding model learns thousands of such dimensions from data instead of us choosing them by hand. Same mechanism, more dimensions — and exactly what powers RAG retrieval.)

3 · Word arithmetic — the analogy trick

Here's the result that made embeddings famous (word2vec, 2013). Because related words end up in related directions, you can do algebra on meaning. The vector from man to woman turns out to be roughly the same as the vector from king to queen — a consistent "gender direction." So if you take king, subtract the male part, and add the female part — king − man + woman — you land almost exactly on queen. Nobody programmed that in; the model discovered these directions purely from how words co-occur in text.

Press the button to compute the result vector (✦) and find its nearest word. This is why embeddings feel like they "understand" — relationships in the world show up as geometry in the space.

Key takeaways

• An embedding turns meaning into a vector — trained so similar things are close.
• Cosine similarity measures direction (angle), ignoring length — the standard way to score "how similar."
• Semantic search = nearest neighbors in embedding space; it matches meaning, not keywords.
• Analogies work because relationships become consistent directions you can add and subtract.
• Where do they come from? A neural network trained on huge text (word2vec, GloVe, or modern sentence-transformers) learns the layout from data — you just use the vectors.