Decision trees

A tree plays twenty questions with your data: "income > ₹6L? age < 30?" — each question slices the space in two, and the slices stack into a staircase decision boundary. Watch a real tree grow, watch impurity fall, and watch it overfit before your eyes.

Gini impuritygreedy splits axis-aligned boundarydepth = capacity

How a tree decides where to cut

Start with all the points in one box. The box is impure — it holds a mix of both classes. The tree tries every possible cut on every feature ("x < 0.31? x < 0.32? … y < 0.5?") and asks the same question of each: how pure do the two children get? Purity is measured by Gini impurity — the chance that two points drawn from the box disagree on class. A 50/50 box scores 0.5 (maximum confusion), an all-one-class box scores 0 (nothing left to ask). The cut that drops impurity the most wins, and the algorithm recurses into each child — greedy, one level at a time, never looking back. That's the whole algorithm (CART). No gradients, no distances — which is also why trees don't care about feature scaling: "x < 5" splits identically whether x is in metres or kilometres.

Grow it one level per beat — and add your own points

click the plot to add a

train acc

test acc

impurity

Watch the train and test bars diverge as depth grows — that widening gap IS overfitting. Points you add become training data: drop a few blues inside green territory and watch the tree fence them in.

Grow slowly and read the three bars. Levels 1–3: every cut buys real purity — both accuracies climb and the boundary starts tracing the true boundary between the classes (a curve, which the tree can only approximate with axis-aligned staircase steps — a tree literally cannot draw a diagonal line). Keep growing past depth 5 and the character changes: the tree starts fencing off individual noisy points in tiny boxes. Train accuracy marches to 100% — the tree can always memorize — while test accuracy stalls and then slides backwards. You are watching the bias–variance tradeoff in its purest form: depth is the capacity dial, and unlimited depth is memorization.

So how deep should it go?

You cap it. max_depth, min_samples_leaf (don't create boxes holding fewer than n points), or grow-then-prune (cut back branches that don't earn their keep on validation data). All three say the same thing: a box containing three noisy points is not knowledge. Pick the cap by cross-validation, like every other hyperparameter. But the deeper truth — the one that leads to the next explainer — is that single trees are inherently high-variance: re-draw the training sample and the greedy cuts land elsewhere, reshaping the whole tree. Averaging many such trees is what makes a random forest.

⚠️ Exam traps: splits are chosen by impurity reduction (Gini or entropy), not randomly, not by correlation · trees need no feature scaling · the boundary is axis-aligned (no diagonals) · an unconstrained tree reaches 100% train accuracy — that's a bug (memorization), not a feature.

Check yourself

Takeaways: tree = recursive twenty-questions; each cut maximizes purity gain · Gini = P(two random points from the box disagree) · greedy, level by level, never revisits · staircase boundaries · depth is the overfitting dial — cap it, prune it, or better: forest it.

Curated companion (the classic scrollytelling take): R2D3 — A Visual Introduction to Machine Learning.