Backpropagation

How a network learns: run inputs forward to a loss, then push gradients backward through the same graph using the chain rule — so every weight learns exactly how it nudged the error. It's just bookkeeping for derivatives.

computation graphchain rule forward & backward passBYO-1 autograd

One neuron, end to end

A single neuron with a sigmoid activation and a squared error: forward computes z = w·x + b, then a = σ(z), then L = (a − y)². It opens on the finished sweep (black = forward value, red = gradient); press ▶ to replay it, one local derivative per step.

w (weight) 0.8the knob training actually turns — we want ∂L/∂w

x (input) 1.5the input: fixed data, but its value SCALES w's gradient

b (bias) -0.5the shift — its gradient is the raw upstream signal

y (target) 1.0the target the loss compares against

How to read it: the black number in each node is its forward value. The red number that appears is the gradient ∂L/∂(node) — how much the loss changes if you wiggle that value. Each red number is the red number after it times a small local derivative written on the edge. That product is the chain rule.

The goal: how should each weight change?

Training is gradient descent: nudge each parameter a little in the direction that reduces the loss. To do that you need the gradient of the loss with respect to every parameter — ∂L/∂w for millions of weights. Computing each one independently would be hopeless. Backpropagation is the algorithm that gets all of them in a single backward sweep, by reusing intermediate results.

The trick is to see the network as a computation graph: a chain of small operations, each of which knows its own local derivative. The chain rule says the derivative along a path is the product of the local derivatives, and when a value feeds several places you sum over paths. Backprop just walks the graph from the loss backward, multiplying local derivatives as it goes.

Why it's done this way

Forward then backward. The forward pass caches each node's value; the backward pass reuses those cached values to compute local derivatives, so the whole gradient costs about the same as one extra forward pass — not one pass per parameter.
Local + chain = global. Every operation only needs to know how to turn the gradient flowing into it into the gradient flowing out (multiply by its local derivative). String these together and you get ∂L/∂w for the deepest weight in a 100-layer net.
Sum at forks. If a value is used in two places, its gradient is the sum of the gradients coming back from both — that's how shared weights and skip connections work.
Then descend. Once you have ∂L/∂w, the update is just w ← w − lr · ∂L/∂w. (See the optimizer comparison for smarter update rules.)

This is exactly what autograd engines (PyTorch's .backward(), TensorFlow's tape) do automatically: they record the graph during the forward pass, then replay it in reverse. Vanishing / exploding gradients come straight from this picture too — multiply many small (or large) local derivatives down a deep chain and the product shrinks toward 0 (or blows up).

Check yourself

Takeaways: backprop computes ∂L/∂w for every parameter in one backward sweep of the computation graph. Forward caches values; backward multiplies local derivatives (chain rule) and sums at forks. It makes training deep networks tractable. Build a working autograd engine from scratch in BYO-1, then a mini-PyTorch in BYO-5.