One deep tree memorizes. A hundred deep trees, each shown a different bootstrap sample and forced to consider different features, memorize different noise — and when they vote, the noise cancels while the signal stacks. That's the whole trick, and it wins on tabular data to this day.
From the trees explainer you know a deep tree is a high-variance learner: re-draw the training sample and its greedy cuts land somewhere else entirely. Usually variance is a disease. Bagging turns it into fuel. Train many trees, each on a bootstrap sample — n rows drawn with replacement, so each tree sees roughly 63% of the data with some rows repeated — and let them vote. Each tree still overfits, but it overfits its own sample's quirks. Where the trees disagree (the noise), the vote washes it out; where they agree (the signal), the vote locks it in. Averaging many high-variance, low-bias estimators keeps the low bias and divides the variance.
One more twist makes it a random forest: at every split, each tree may only choose among a random subset of features (√p of them, typically). Without this, every tree would grab the same single dominant feature for its first cut and the "crowd" would be a hundred copies of one opinion — averaging correlated errors cancels nothing. Feature randomness decorrelates the trees; bootstrap + feature subsets together manufacture the disagreement that makes the vote work.
the forest's vote
the LAST tree added (alone)
Each tree is deep (depth 7) and individually overfit — add a few and compare the two panels.
Look at the right panel: each individual tree draws a jagged, paranoid boundary — tiny boxes, weird peninsulas, classic overfitting. Now look left: as the votes accumulate, the forest's boundary smooths out into something none of its members drew. By ~20 trees the forest's test accuracy sits comfortably above the average single tree, and adding more trees never hurts — the vote only stabilizes. (More trees costs compute, not accuracy: unlike depth, n_estimators is not an overfitting dial.)
Because each tree skipped ~37% of the rows, those rows are honest validators for that tree: score every point using only the trees that never saw it and you get the out-of-bag (OOB) estimate — cross-validation-quality feedback with zero extra training. And by tallying how much each feature's splits reduce impurity across the whole forest, you get feature importances for free — crude but priceless for a first conversation with any dataset. The forest's real limitation is the flip side of its strength: averaging fights variance, not bias — if a single tree can't represent the pattern, a thousand can't either. That's the gap boosting (sequential error-correction: AdaBoost, XGBoost) attacks from the other side: bagging = parallel + variance↓, boosting = sequential + bias↓.
Curated companion: MLU-Explain — Random Forest.