How an LLM Picks the Next Token

A language model doesn't "choose a word" — it outputs a probability over every token and then you sample from it. temperature, top-k, and top-p are the three knobs that reshape that distribution before the dice are rolled.

softmax over logitstemperature top-ktop-p / nucleusgreedy vs sampling

The model gives you a distribution, not an answer

At each step the model produces one raw score (a logit) for every token in its vocabulary. A softmax turns those scores into probabilities that sum to 1 — a full distribution over "what could come next." Decoding is the policy for turning that distribution into an actual token.

The simplest policy is greedy: always take the single highest-probability token (argmax). It's deterministic and fine for short factual answers, but it's also bland and prone to loops ("the the the"). The alternative is to sample — roll a weighted die — which brings variety and creativity, at the cost of occasionally picking something odd. The three knobs below all exist to control how much of the tail you let into that die roll. Take the prompt "The cat sat on the ___":

temperature 1.00reshapes the whole distribution: <1 sharpens toward the favorite, >1 flattens toward chaos

top-k 0hard cutoff: only the k most-likely tokens may be sampled (0 = off)

top-p 1.00adaptive cutoff: smallest set of tokens covering p of the probability (nucleus)

How to read it: each bar is a candidate next token; its length is the final probability the model would sample from after your settings. Solid green bars are in play; faded bars were cut by top-k / top-p and have zero chance. The dashed line marks the top-p nucleus boundary. The small count on the right is how often each token actually came up when you sampled.

Knob 1 — temperature: sharpen or flatten

Temperature divides the logits before the softmax. Low temperature (<1) sharpens the distribution — the leader's probability balloons toward 1, so it behaves almost like greedy. High temperature (>1) flattens it — long-shot tokens get a real chance, so output gets more surprising (and eventually incoherent). As T → 0 you recover pure greedy; at T = 1 you sample from the model's honest distribution. Slide it and watch every bar grow or shrink together.

Knob 2 — top-k: keep only the k best

Top-k truncates the distribution to the k highest-probability tokens, throws away the rest, and renormalizes. It's a hard cap on how weird the next token can be — with k = 1 you're back to greedy; with k = 5 the model can only ever pick from its top five guesses. The weakness: a fixed k is sometimes too generous (when the model is very confident, even the 2nd choice is junk) and sometimes too strict (when many tokens are genuinely plausible). (Set the slider to 0 to turn top-k off.)

Knob 3 — top-p (nucleus): keep just enough probability mass

Top-p fixes top-k's rigidity by being adaptive. Instead of a fixed count, you keep the smallest set of tokens whose probabilities add up to at least p (say 0.9), then sample from that "nucleus." When the model is confident, one or two tokens already cover 90% of the mass, so the nucleus is tiny and the output stays safe. When the model is genuinely unsure, the mass is spread out, so the nucleus grows and allows more variety. That's why top-p ≈ 0.9 with temperature ≈ 0.7–1.0 is the most popular default — it adapts to the model's own confidence.

Which to reach for

Factual / code / extraction: low temperature (0–0.3) or greedy — you want the most likely, reproducible answer, not creativity.
Chat / general writing: the common default temperature ≈ 0.7, top-p ≈ 0.9 — coherent but not robotic.
Brainstorming / poetry: higher temperature (1.0–1.3) and/or larger top-p for more surprising results — accept the occasional misfire.
Reproducibility: set temperature to 0 (greedy) — and remember even then, infrastructure can introduce tiny nondeterminism.

You rarely tune all three at once: pick temperature for how adventurous, and one of top-k / top-p (usually top-p) as a safety net against the genuinely bad tail.

Takeaways: the model emits a probability distribution; decoding samples from it. Temperature sharpens (<1, toward greedy) or flattens (>1, more random) the whole distribution. Top-k keeps a fixed number of top tokens; top-p keeps the smallest set covering probability p and adapts to the model's confidence. Greedy = temperature 0.