Optimizers, Head to Head

Every optimizer is a recipe for turning a gradient into a step. On a stretched "ravine" loss surface their personalities show: plain SGD zig-zags, momentum builds speed, and the adaptive methods (RMSprop, Adam) march almost straight to the bottom.

SGDmomentum RMSpropAdamlearning rate

The same gradient, four different steps

Gradient descent says "step downhill," but how far and using what memory is the optimizer's job. The challenge is that real loss surfaces are badly scaled: steep in some directions, nearly flat in others (a "ravine"). A single global learning rate is then a no-win — large enough to make progress along the flat axis, it overshoots and oscillates along the steep one. The four classic optimizers each fix this differently:

SGD — θ ← θ − lr·g. Pure downhill, fixed step. In a ravine it bounces back and forth across the steep walls while crawling along the floor.
Momentum — accumulate a velocity (v ← βv − lr·g; θ ← θ + v). Like a ball rolling downhill, it cancels the back-and-forth oscillations and accelerates along the consistent direction.
RMSprop — keep a running average of each coordinate's squared gradient and divide by its root. This gives every direction its own adaptive step, shrinking the steep axis and enlarging the flat one — so the ravine looks round again.
Adam — momentum + RMSprop together (a velocity and per-coordinate scaling), plus a bias correction. The robust default that usually "just works."

Watch them on f(x,y) = ½(x² + 20y²) — gentle along x, 20× steeper along y. Same start, same base learning rate. Press ▶ — then pause anywhere, drag the timeline back, or step the race one move at a time. The update rule that moved the farthest this step lights up in the code panel:

learning rate 0.08

Same start (gray dot), same learning rate. The whole race is precomputed the moment you move the slider — so the timeline can be scrubbed freely: rewind to watch SGD's first overshoot again, or step frame by frame.

Reading the race & choosing one

You'll see SGD trace a slow zig-zag, momentum swing wide then settle, and RMSprop / Adam cut nearly straight to the center because they rescale the steep y axis. Nudge the learning rate up and watch SGD become unstable (it diverges along y) long before the adaptive methods do — that's the practical reason adaptive optimizers are forgiving.

Default to Adam (often lr ≈ 1e-3) for deep nets — it's robust to bad scaling and needs little tuning. AdamW (Adam + decoupled weight decay) is the modern standard for transformers.
SGD + momentum still wins some vision benchmarks on final generalization, at the cost of more careful learning-rate scheduling.
The learning rate is still the most important knob regardless of optimizer — too high diverges, too low crawls. Pair any of these with a schedule (warmup + decay) in practice.

(This is the discrete cousin of Distill's "Why Momentum Really Works" — open it for the continuous intuition.)

Check yourself

Takeaways: all optimizers turn gradients into steps, differing in memory and per-direction scaling. Momentum smooths oscillations; RMSprop adapts the step per coordinate; Adam combines both and is the safe default. Bad surface scaling is why a single fixed learning rate struggles — and why adaptive methods are more forgiving of the learning-rate choice.