The Illustrated RAG Pipeline

Retrieval-Augmented Generation, end to end — type a question and watch it chunk → embed → retrieve → ground → generate, fully offline, no API calls.

retrievalembeddings cosine similarityBYO-9 RAG engine

The problem RAG solves

A large language model is, at heart, a very sophisticated autocomplete: it predicts the next word using patterns frozen into its weights during training. That gives it two stubborn limitations. First, its knowledge is frozen and generic — it knows nothing about your company's wiki, last night's incident report, or a PDF you just downloaded. Second, when it doesn't know something, it often makes up a fluent, confident, wrong answer (a "hallucination"), because generating plausible text is exactly what it was trained to do.

You could re-train the model on your documents, but that's slow, expensive, and goes stale the moment your data changes. Retrieval-Augmented Generation (RAG) takes a cheaper, smarter route: instead of forcing the model to memorize your knowledge, you let it look things up at the moment it answers.

The open-book exam analogy. A closed-book exam tests what you memorized — and you'll bluff when you forget. RAG turns it into an open-book exam: right before answering, you flip to the most relevant pages and write your answer from them. The model's reasoning ability stays; the facts come from your documents. That's why a good RAG answer is both fluent and grounded — and can point to its sources.

How it works, end to end

RAG is two halves glued together. Retrieval finds the passages most relevant to the question; generation writes an answer using only those passages. The trick that makes retrieval work is embeddings: every passage (and the question) is converted into a vector of numbers so that text with similar meaning ends up close together in vector space. We then rank passages by how close they are to the question — usually with cosine similarity (the angle between two vectors). The closest few get pasted into the prompt as context.

Below is a tiny but real RAG engine running entirely in your browser — a bag-of-words embedder, cosine search, and a templated generator (no API calls). Type a question and watch each stage light up; the same five stages run in every production RAG system, just with stronger components.

The pipeline

1 · Your question

Ask the knowledge base

top-k retrieved 2

Try: “what is a vector database?”, “what is prompt injection?”, “why split documents?”

Step 2, in depth — turning meaning into numbers

This is the heart of RAG, so it's worth slowing down. To find "relevant" passages we need a way to measure relevance mathematically. The idea: convert each passage into a list of numbers (a vector, or "embedding") chosen so that passages about similar things have similar vectors. Real systems use a trained neural network (like sentence-transformers) for this; our toy uses a simpler bag-of-words vector — one slot per vocabulary word, counting how often it appears — which is enough to see the mechanics.

To compare two vectors we use cosine similarity: the cosine of the angle between them. It runs from 1.0 (pointing the same way → very similar) through 0 (perpendicular → unrelated). We use the angle, not the distance, so a long document and a short query can still match if they point the same direction. We embed the question, compare it to every passage, and keep the top-k highest-scoring ones. (Curious about the geometry? The embeddings explainer lets you drag vectors and watch cosine change.)

One detail hides a real tradeoff: chunking. Documents are split into small passages before embedding. Chunks too large dilute the signal (one vector tries to represent many ideas); too small and you lose context. Chunk size and overlap are among the biggest levers on RAG quality.

2 · Knowledge base → embed → retrieve (cosine similarity)

Each passage is embedded into a vector; we rank passages by cosine similarity to the question. Retrieved passages are highlighted.

Step 3, in depth — grounding the model

Now we do something almost embarrassingly simple: we paste the retrieved passages straight into the prompt, with an instruction like "Answer using only the context below, and cite your sources." The model never "knew" these facts — it's reading them right now, the way you'd read the page you just flipped to in an open-book exam. This single move is what makes RAG answers grounded: the facts come from your documents, and instructing the model to stick to them sharply reduces hallucination.

The numbered tags [1], [2] aren't decoration — they let the model (and the reader) cite exactly which passage supports each claim, so a human can verify the answer instead of trusting it. In regulated domains (finance, security, medicine) this traceability is often the whole point. Watch the prompt rebuild as you change the question or top-k.

3 · Grounded prompt (what the LLM actually sees)

4 · Generated answer (with citations)

What RAG is good at — and where it breaks

RAG shines when answers should come from a specific, changing, or private body of knowledge: company docs, product manuals, legal/financial filings, a security knowledge base. You can update the knowledge by changing documents — no retraining — and every answer can cite its evidence.

But RAG is only as good as its retrieval. If the right passage isn't retrieved, the model can't use it — "garbage in, garbage out." Common failure modes and their fixes:

Wrong chunks retrieved → tune chunk size/overlap; add a re-ranker that re-scores the top candidates more carefully.
Vocabulary mismatch (question and answer use different words) → real embedding models handle this far better than our bag-of-words toy; you can also rewrite/expand the query first.
The answer spans many passages → retrieve more, or summarize retrieved context before answering.
The model ignores the context and bluffs anyway → stronger "use only the context" instructions, and evaluation (e.g. RAGAS faithfulness) to catch it.

Check yourself

The big picture: the model never "memorized" your docs — it just read the retrieved context. Swap the bag-of-words embedder for sentence-transformers, the in-memory search for a real vector database, and the templated generator for an LLM, and you have a production RAG app. Build exactly that, piece by piece, in BYO-9, and see the retrieval internals in the vector-search explainer.