Activation functions

The small non-linear function applied after every neuron — and the reason a deep network can learn anything more interesting than a straight line. Pick the wrong one and your gradients vanish; this is the knob behind several exam questions.

sigmoidtanh ReLUsoftmax vanishing gradient

Why a non-linearity is non-negotiable

A neuron computes a weighted sum z = w·x + b — that's linear. Stack two linear layers and you get… still just a linear function (a composition of lines is a line). No matter how deep, the network could only draw straight boundaries. The activation function squeezes a bend into each neuron, and that is what lets stacked layers compose into arbitrarily curved functions. So every activation here is non-linear — including ReLU, which looks like two straight pieces but has a kink, and a kink is enough.

The blue curve is the function; the amber dashed curve is its derivative — the slope gradients flow through during backprop. Watch where the derivative goes flat:

activation

How to choose

ReLU — the default for hidden layers. Cheap, and its derivative is exactly 1 for positive inputs, so gradients don't shrink as they pass through — that's why it largely solved the vanishing-gradient problem and made deep nets trainable. Downside: neurons stuck at negative inputs output 0 and can "die"; Leaky ReLU lets a small slope through to avoid that.
Sigmoid — squashes to (0, 1), so it reads as a probability. Use it for the output of a binary classifier. Avoid it in hidden layers: its derivative is ≤ 0.25 and ≈ 0 at the tails, so stacking sigmoids multiplies tiny gradients into nothing (the vanishing-gradient problem).
tanh — like sigmoid but centred at 0, range (−1, 1); often better than sigmoid in hidden layers, but still saturates at the tails.

Softmax: the multi-class output

For multi-class output you need a vector of probabilities that sum to 1. Softmax takes raw scores (logits), exponentiates, and normalises: softmax(z)ᵢ = e^{zᵢ} / Σ e^{zⱼ}. The output is a proper probability distribution — every value in (0, 1) and the whole vector adding to 1. Drag the three logits and watch the distribution respond (bigger gap ⇒ more peaked):

z₁

z₂

z₃

Takeaways: activations add the non-linearity that lets depth represent curved functions (ReLU's kink counts — it is non-linear). ReLU is the hidden-layer default (slope 1 ⇒ no vanishing gradient); sigmoid/tanh saturate and vanish, but sigmoid is right for a binary output. Softmax turns logits into a multi-class distribution in (0,1) that sums to 1.

Curated companions: TensorFlow Playground (swap activations live) · CS231n notes.