How modern frontier models get enormous parameter counts without an enormous compute bill: replace one big feed-forward block with many smaller "experts" and a router that sends each token to only a couple of them. You pay for what you use, not for the whole bank.
In a normal Transformer layer, every token passes through the same feed-forward network — all of its parameters run for every token. An MoE layer swaps that single network for a set of N experts (each its own small FFN) plus a tiny router. For each token, the router scores the experts and sends the token to only the top-k of them (often k = 2). The chosen experts run, their outputs are blended by the router's weights, and the rest sit idle. That's conditional computation: the model holds a huge number of parameters, but only a small fraction activates per token.
Watch four tokens get routed. Each line goes to a token's selected experts (thicker = higher gate weight); dimmed experts are skipped for that token:
All N experts sit in memory as a stack (that's the serving cost); the green ones are the k that actually ran this batch — the compute cost. Sparsity is the gap between the two.
The payoff: a model with, say, 8 experts has roughly 8× the FFN parameters (more knowledge, more capacity) but if each token uses only 2, it does about the same compute per token as a dense model one-quarter the size. That's how you get a "huge" model that's cheap to run — total parameters and active parameters become two different numbers.
It isn't free. The router must spread tokens reasonably evenly — if everyone piles into one popular expert, you get a bottleneck and wasted capacity, so MoEs add a load-balancing loss to keep usage even. And all N experts must live in memory even though only k run, so MoEs are memory-hungry to serve even when they're cheap to compute. The routing decision is also discrete, which makes training a bit trickier than a plain dense layer.
Curated companions: Hugging Face — Mixture of Experts Explained · bbycroft — LLM internals.