Byte-Pair Encoding (BPE) Tokenization

Before a language model sees a single number, your text is chopped into tokens. BPE is the algorithm that decides where the cuts go — by learning, from data, which character sequences are worth gluing together.

subword tokensmerge rules vocab size vs lengthBYO-7 tokenizer

Why models need tokens at all

A neural network can't consume raw text — it consumes vectors. So step one of every LLM is a lookup table that maps each token to an integer ID and then to a learned vector (an embedding). That means we need a fixed, finite vocabulary. Two obvious choices both fail:

One token per word — the vocabulary would be effectively infinite (every name, typo, URL, and compound word), and any word unseen in training becomes a useless <UNK>.
One token per character — a tiny vocabulary, but now "tokenization" is 12 tokens instead of ~2, so sequences get long and the model wastes capacity relearning that t-h-e is "the".

Subword tokenization is the compromise everyone settled on: common words become a single token, rare words break into a few meaningful pieces, and any string is representable (worst case, down to bytes). BPE is the most common way to learn that subword vocabulary.

The BPE idea: greedily merge the most frequent pair

BPE starts with the most granular units — individual characters — and then repeats one move: scan the whole corpus, find the adjacent pair of symbols that occurs most often, and merge it into a single new symbol. Do that N times and you've learned N merge rules. Frequent sequences (e+s → es, then es+t → est) get welded into reusable chunks, while rare words simply never trigger a merge and stay in pieces.

Below is BPE training, live. It starts on the classic toy corpus, but the whole point is the input box: type any text you like — a sentence, a tongue-twister, code — and the entire training run is recomputed instantly. Press ▶ to watch merges happen one by one, scrub the timeline back and forth, and watch the matching line of the algorithm light up. The end-of-word marker · keeps word boundaries honest:

Corpus — each word as its current token sequence:

Next pair to merge (by frequency):

Learned merge rules (the vocabulary grows):

How to read it: every word is a row of symbol-chips. The highlighted pair is the most frequent adjacent pair in the whole corpus (counts weighted by each word's frequency). Merging it adds one rule to the vocabulary and shortens the words that contained it. Notice newest (frequency 6) drives its pieces to merge first — frequency is everything.

Using the learned rules to tokenize new text

Once the merges are learned, tokenizing is mechanical: split a word into characters, then apply the merge rules in the order they were learned, gluing pairs wherever they appear, until no rule applies. A word made of frequent pieces collapses to a few tokens; an unusual word stays in smaller chunks. Type a word below — and here's the fun part: scrub the training timeline above and watch the same word tokenize differently as the vocabulary grows rule by rule:

word to tokenize

applying the learned merges in order:

Why tokenization quietly shapes everything

Cost & context are measured in tokens. API pricing and the context window are token counts, not words. Code, JSON, and many non-English languages tokenize into more tokens per character, so they're more expensive and eat the context budget faster.
It explains classic LLM "failures." "How many R's in strawberry?" is hard because the model sees a couple of subword tokens, not the letters. Arithmetic and rhyming wobble for the same reason — the unit of perception isn't the character.
Real tokenizers work on bytes, not letters. GPT-style models use byte-level BPE: the starting alphabet is the 256 byte values, so any Unicode text, emoji, or binary is representable with zero <UNK> tokens — the toy above just uses letters to stay readable.
The vocabulary is frozen after training. Merge rules are learned once on a big corpus and shipped with the model; at inference you only ever apply them (as in the box above).

Check yourself

Takeaways: BPE learns a subword vocabulary by repeatedly merging the most frequent adjacent pair, balancing a small vocabulary against short sequences. Frequent words become one token; rare words split into pieces; nothing is unrepresentable. Tokens — not words or characters — are the LLM's true unit, which drives cost, context limits, and a whole class of quirky failures. Build a BPE tokenizer end-to-end in BYO-7.