Regularization: Ridge & Lasso

Left alone, a model will chase noise — fitting big, wobbly coefficients that overfit. Regularization adds a penalty on coefficient size to the loss, keeping the model simpler. The two classic penalties, L2 (Ridge) and L1 (Lasso), look almost identical but do something profoundly different to your features.

L2 / RidgeL1 / Lasso shrinkagesparsity feature selection

Penalise big coefficients

You fit a model by minimising error. Regularization adds a second term — a penalty on how large the coefficients are — so the model must justify every bit of complexity:

Ridge (L2): minimise error + λ·Σ wᵢ² — penalises the squared size.
Lasso (L1): minimise error + λ·Σ |wᵢ| — penalises the absolute size.

The strength λ dials how hard you push. Crank it up and watch the two penalties shrink the same starting coefficients in very different ways:

regularization strength λ

Ridge (L2) — shrinks all toward 0, smoothly

Lasso (L1) — drives some to exactly 0

Why L1 zeros things out and L2 doesn't

This is the famous picture. The penalty defines a "budget" region the coefficients must stay inside, and the solution is where the error contours first touch it. L2's budget is a circle (smooth) — contours usually touch it at a point where both coefficients are small but non-zero. L1's budget is a diamond with sharp corners on the axes — and contours tend to touch a corner, where one coefficient is exactly zero.

That single geometric difference is why Lasso performs feature selection — it throws features out entirely, giving a sparse, interpretable model — while Ridge keeps every feature but small, which behaves better when features are correlated. (Elastic Net blends both.)

Takeaways: regularization adds a coefficient-size penalty (strength λ) to fight overfitting. Ridge/L2 (Σwᵢ²) shrinks all coefficients smoothly toward — but not to — zero; Lasso/L1 (Σ|wᵢ|) drives some to exactly zero, doing automatic feature selection. The diamond's corners on the axes are why. Use Lasso for sparsity/interpretability, Ridge for correlated features.

Curated companion: StatQuest — Ridge & Lasso · ISLP Ch.6.