Cross-validation

One train/test split gives you one number — and that number can be lucky. K-fold cross-validation gives you a distribution: every point takes a turn being the exam, and the spread across folds tells you how much to trust the mean.

k-foldmean ± std split lucktime-series traps

The problem: split luck

You split your data 80/20, train, and score 0.86. Ship it? Here's the uncomfortable question: if you'd cut the data at a different random spot, would you have gotten 0.86 again — or 0.79, or 0.91? With one split you cannot know. Small datasets especially can hand you a test set that's accidentally easy (or brutal), and your single score is really a sample of size one from a distribution you never looked at.

Cross-validation looks at the distribution. Chop the data into k folds. Train on k−1 of them, validate on the one held out — then rotate so each fold gets exactly one turn as the validator. You get k scores. Report the mean ± standard deviation: the mean is your best estimate, the std is your honesty about it. "0.82 ± 0.02" and "0.82 ± 0.14" are the same headline and completely different stories — the second one says "this model's performance depends heavily on which data it sees; don't trust me."

Watch k-fold rotate

k (folds)

Each row is one round: the amber blocks train, the green block validates, and the score lands on the right. No point is ever validated by a model that trained on it — and every point gets validated exactly once. Try k=2 (big validation sets but only two scores — coarse) versus k=10 (ten scores, but each validation set is tiny and noisy). k=5 or 10 is the boring, correct default. The extreme, k=n, is leave-one-out: n rounds of training for one prediction each — statistically neat, computationally silly for most datasets.

What cross-validation is actually for

Three jobs, in increasing order of importance. One: a more reliable score than any single split. Two: a variability estimate — the ±std that single splits can't give. Three, the big one: a safe playground for choosing things. Every time you compare models, tune a hyperparameter, or pick features, you're selecting — and selection slowly overfits whatever data does the scoring. If that data is your CV folds, fine: you can always cut fresh folds. If it's your test set, you just spent the only honest number you had. Tune on folds; touch test once, at the very end.

⚠️ The two classic leaks: ① preprocessing fit on ALL data before splitting (the scaler saw the test fold's statistics — use a Pipeline so each fold refits it); ② shuffled time series — random folds put future days in training, and the model "remembers tomorrow". Demo below.

The time-series trap

Takeaways: one split = one lucky/unlucky draw; k-fold = mean ± std you can trust · every point validates exactly once · tune hyperparameters on folds, spend the test set once · Pipelines make CV leakage-proof · time series must be split in time order (walk-forward), never shuffled.

Curated companion: MLU-Explain — Cross-Validation.