LSTM & GRU Gates

Plain RNNs forget. LSTMs add a protected memory cell — a conveyor belt of state — with gates that decide what to forget, what to store, and what to reveal. Drive the gates by hand and watch a memory persist or vanish.

cell stateforget / input / output gates long-range memoryGRU

The problem: vanilla RNNs can't hold on

A recurrent network reads a sequence one step at a time, carrying a hidden state forward. In a plain RNN that state is rewritten every step by a squashing function, so information from far back gets multiplied away to nothing — the vanishing gradient problem. The network literally can't connect "the keys… (20 words) … are on the table" because the subject faded long before the verb.

The LSTM (Long Short-Term Memory) fixes this with a separate cell state c that runs straight down the sequence like a conveyor belt, touched only by simple, mostly-linear operations. Because information can ride the belt almost unchanged, gradients survive across many steps. What goes on and off the belt is controlled by three gates — each a value in [0,1] (from a sigmoid) acting as a dial for "how much to let through."

The three gates

At each step the cell updates as c_t = f · c_{t−1} + i · g̃ and outputs h_t = o · tanh(c_t), where:

Forget gate f — how much of the old memory to keep (1 = keep all, 0 = erase).
Input gate i — how much of the new candidate g̃ to write onto the belt.
Output gate o — how much of the (squashed) cell state to expose as this step's output h_t.

Set the gates and the incoming candidate, then step through time and watch the cell state (the belt) and the output evolve:

forget gate f 1.00–1 valve on the OLD memory: 1 = keep everything, 0 = wipe

input gate i 0.00–1 valve on NEW information: how much of the candidate gets written

output gate o 1.00–1 valve on the readout: how much of the cell state shows in the output h

candidate g̃ 0.8the new content on offer this step (tanh-squashed to −1…1)

presets:

The intuition the presets show

Remember (f=1, i=0): the belt carries the old value untouched — this is the whole point. With the forget gate open and the input gate shut, memory (and gradient) flows across arbitrarily many steps without decaying.
Overwrite (f=0, i=1): wipe the old memory and write the new candidate — useful at a sentence boundary or a "reset" token.
Accumulate (f=1, i=1): keep the old and add the new — the cell acts like a running sum / counter.
Forget (f=0, i=0): clear the belt back toward zero.

In a real LSTM you don't set these by hand — each gate is a tiny learned layer (σ(W·[h_{t−1}, x_t] + b)) that computes its dial from the current input and previous output. Training teaches the gates when to remember and forget; here you're playing the role of those learned layers.

GRU — the streamlined cousin

The GRU (Gated Recurrent Unit) keeps the gating idea but with fewer parts: it merges the cell and hidden state into one, and uses just two gates — an update gate (how much to replace the state, combining LSTM's forget+input into one dial) and a reset gate (how much past state to use when proposing the new candidate). Fewer parameters means it's a bit faster and often matches the LSTM on smaller datasets; the LSTM's extra flexibility can edge ahead on larger ones. Both have largely been superseded by attention/transformers for long sequences, but the gating principle — multiplicative dials protecting a memory path — is a foundational idea worth owning.

Takeaways: LSTMs add a protected cell-state conveyor belt so memory and gradients survive long sequences. Three sigmoid gates dial it: forget (keep old), input (write new), output (reveal). c_t = f·c_{t−1} + i·g̃; h_t = o·tanh(c_t). The GRU does the same job with two gates and one state. Gates are learned, not hand-set.