Before transformers: recurrent networks

Read this sentence one word at a time: The keys to the cabinet in the hall by the door … When you reach the end, you know the next word should be “are”, not “is”. Somewhere in your head you kept track of the fact that the subject was plural, through nine words that had nothing to do with it.

The transformers in the rest of this track solve that problem by looking back at every earlier word directly. Before 2017, the leading approach was different: read the words in order and carry a running summary forward. This lesson is about those networks, what made them hard to train, and why attention replaced them.

A network with a loop

A neural networkNeural networkA model built from layers of artificial neurons, where each layer's outputs become the next layer's inputs. Training adjusts all of the weights together.Taught in Networks of neuronsOpen in glossary layer takes a fixed number of inputs. Sentences come in every length. A recurrent neural networkRecurrent neural networkA network that reads a sequence one step at a time, applying the same weights at every step and passing a hidden state from each step to the next.Open in glossary (RNN) handles any length by applying the same small network once per word, and feeding each step’s output back in as an input to the next step.

What it feeds back is a vector called the hidden stateHidden stateThe vector a recurrent network carries from one step of a sequence to the next. It is the network's only memory of the inputs it has read so far.Open in glossary, $h_t$. At step $t$ the network combines the previous hidden state with the current input $x_t$:

The weights $W$, $U$, and $b$ are the same at every step. The hidden state starts at zero and is the network’s only memory: everything it knows about the words so far must fit into this one vector. To use it as a language model, a final layer turns each $h_t$ into logits over the vocabulary, exactly as in the previous lesson, and the network predicts the next token at every step.

Elman (1990) trained small networks of this kind, now often called Elman networks or simple recurrent networks, to predict the next letter or word. Their prediction errors rose at the start of each new word, and their hidden states grouped words into categories such as nouns and verbs, though they were never told about words or categories.

Watch a hidden state carry a signal

The demo shrinks this to the smallest possible case: a hidden state of one number. The subject word adds $+1$ for one key or $-1$ for several; every other word adds 0. At the blank, a positive state means “is” and a negative one means “are”. With a single number, the matrix $W$ becomes a single recurrent weight $w$, so you can see exactly what it does.

import numpy as np

# The demo's long sentence: "The key that my grandmother hid ... last winter ___".
# Input is +1 at the singular subject ("key") and 0 for every other word.
x = np.zeros(20)
x[1] = 1.0

def run(w, u=1.0, cell="tanh"):
    h, states, factors = 0.0, [], []
    for xt in x:
        if cell == "tanh":
            h = np.tanh(w * h + u * xt)
            factors.append(w * (1 - h**2))  # d h_t / d h_(t-1)
        else:  # no squashing
            h = w * h + u * xt
            factors.append(w)
        states.append(h)
    return np.array(states), np.array(factors)

def gradient_to(step, factors):
    # Backpropagation through time: multiply the factors of every later step.
    return np.prod(factors[step + 1:])

for cell, w in [("tanh", 0.5), ("tanh", 1.0), ("tanh", 1.5), ("linear", 1.5)]:
    states, factors = run(w, cell=cell)
    print(f"{cell:6} w={w}: state at blank {states[-1]:8.4f}, "
          f"gradient reaching 'key' {gradient_to(1, factors):.2e}")

# Gated cell: the input gate opens only at the subject, the forget gate is f.
f = 0.97
c = 0.0
for xt in x:
    c = f * c + (1.0 if xt != 0 else 0.0) * xt
print(f"gated  f={f}: state at blank {np.tanh(c):8.4f}, "
      f"gradient reaching 'key' {f ** 18:.2e}")

tanh   w=0.5: state at blank   0.0000, gradient reaching 'key' 3.17e-06
tanh   w=1.0: state at blank   0.2679, gradient reaching 'key' 4.12e-02
tanh   w=1.5: state at blank   0.8586, gradient reaching 'key' 8.18e-08
linear w=1.5: state at blank 1477.8919, gradient reaching 'key' 1.48e+03
gated  f=0.97: state at blank   0.5212, gradient reaching 'key' 5.78e-01

With $w$ below 1 the memory leaks away: by the end of the long sentence nothing is left and the network cannot tell “is” from “are”. With $w$ above 1 the state settles at a stable value, a little like a switch that stays flipped, and the memory survives. Here the weights are set by hand. A real network has to learn them, and that is where the trouble starts.

Backpropagation through time

To train an RNN, you unroll the loop: draw one copy of the network per step, each passing its hidden state to the next. The result is an ordinary deep network, as deep as the sequence is long, in which every layer shares the same weights. Run backpropagationBackpropagationHow a network works out which way to nudge every weight: one backward pass carries the error from the output back through the layers, giving each weight its share of the blame (its gradient, found with the chain rule).Taught in BackpropagationOpen in glossary on it and add up the gradient each copy contributes to the shared weights. This is backpropagation through timeBackpropagation through timeTraining a recurrent network by unrolling it into one copy per step, running ordinary backpropagation, and adding up the gradients each copy contributes to the shared weights.Open in glossary, set out in its standard form by Werbos (1990).

The gradient that tells the network how the state at step $t$ affected a loss at a later step $T$ passes back through every step in between. By the chain ruleChain ruleThe calculus rule for nested functions: if L depends on a and a depends on z, then dL/dz = dL/da times da/dz. Backpropagation applies it over and over.Taught in BackpropagationOpen in glossary it is a product:

In the demo’s one-number network, each factor is just $w\,(1 - h_k^2)$. The gradient column shows the product, one step at a time, on a logarithmic scale.

Vanishing and exploding gradients

A product of many factors is fragile. If the factors are mostly below 1, it shrinks exponentially: 0.8 to the power 30 is about 0.001. If they are mostly above 1, it grows exponentially. These are the vanishing gradientVanishing gradientA gradient that shrinks toward zero as it is passed back through many steps or layers, because it is multiplied by many factors smaller than 1. Early steps then barely learn.Open in glossary and exploding gradientExploding gradientA gradient that grows very large as it is passed back through many steps or layers, because it is multiplied by many factors larger than 1. Training steps then jump wildly.Open in glossary problems. Hochreiter analyzed them in his 1991 diploma thesis, and Bengio, Simard, and Frasconi (1994) showed that they make long-range dependencies hard to learn by gradient descent.

Bengio et al. also pointed out the dilemma you met in the demo. To store a bit of information reliably, despite noise in the inputs, the network’s state has to sit in a stable region where small disturbances die out. But “small disturbances die out” is the same thing as “the gradient vanishes”. At $w = 1.5$ the network remembers the subject perfectly well, yet the gradient reaching the subject is less than one ten-millionth: training gets almost no signal about which early word caused a mistake at the end.

Gates: a memory that is added to

Long short-term memoryLong short-term memoryA recurrent network whose memory cell is updated by adding gated new information to a gated copy of the old cell, so information and gradients can survive many steps. Introduced in 1997.Open in glossary networks (Hochreiter and Schmidhuber, 1997) attack the vanishing gradient directly. Next to the hidden state they keep a cell state $c_t$, and they change it by addition rather than by squashing it through a weight matrix:

Here $g_t$ is a proposed update computed from the input and the previous hidden state, and $\odot$ multiplies element by element. The gates $f_t$, $i_t$, and $o_t$ are vectors of numbers between 0 and 1, each computed by its own small layer with a sigmoid, from the current input and the previous hidden state:

• The forget gate $f_t$ decides how much of the old cell to keep.
• The input gate $i_t$ decides how much of the new proposal to write.
• The output gate $o_t$ decides how much of the cell to show as the hidden state.

The direct path from $c_{t-1}$ to $c_t$ is multiplied only by $f_t$. When the network learns to hold $f_t$ near 1, both information and gradients pass through many steps nearly unchanged. The demo’s gated cell is this idea with the gates frozen: the input gate opens only on the subject, and you set the forget gate yourself. The original 1997 LSTM had no forget gate; its cell kept everything, which Hochreiter and Schmidhuber called a constant error carousel. Gers, Schmidhuber, and Cummins (2000) added the forget gate so a cell could learn to clear itself.

The gated recurrent unitGated recurrent unitA simpler gated recurrent network than the LSTM, with an update gate that blends the old hidden state with a new candidate and a reset gate. Introduced by Cho et al. in 2014.Open in glossary (GRU) of Cho et al. (2014) is a lighter variant. It has no separate cell: an update gate blends the old hidden state with a new candidate, and a reset gate decides how much of the old state the candidate may use. With two gates instead of three it has fewer parameters, and in practice the two often perform comparably.

The cost of going one step at a time

Gates fixed much of the gradient problem, but not the loop itself. To compute $h_{100}$ you need $h_{99}$, which needs $h_{98}$, and so on. During training the whole sentence is known in advance, yet the network still has to process it in order. A GPU is fast because it does thousands of multiplications at once; an RNN gives it one small step at a time per sequence. Vaswani et al. (2017) put it directly: this sequential nature “precludes parallelization within training examples”.

The fixed-size state was a second bottleneck. In translation, an encoder RNN read the source sentence and a decoder RNN wrote the translation (Sutskever, Vinyals, and Le, 2014). Everything the decoder knew about the source had to pass through one vector. Bahdanau, Cho, and Bengio (2014) let the decoder look back at all the encoder’s hidden states at every step, weighting them by relevance. That was attentionAttentionA mechanism that lets each token build a new vector as a weighted mix of other tokens' value vectors, with weights set by how well its query matches their keys.Taught in Attention, step by stepOpen in glossary, added on top of recurrent networks.

The transformer (Vaswani et al., 2017) took the last step and removed the recurrence entirely. Every position attends directly to every earlier position, so:

• Training is parallel. All positions of a training sequence are computed at once, with a few large matrix multiplications per layer.
• Paths are short. Information from word 1 reaches word 100 in a single attention step, not through 99 multiplications, so its gradient does not have to survive a long product.

The price is that attention compares every pair of positions, so its cost grows with the square of the sequence length, and generation is still one token at a time. You will see both costs, and how the KV cache reduces the second, later in this track. Recurrent ideas have also returned: state space models such as Mamba (Gu and Dao, 2023) keep a fixed-size state, like an RNN, but are designed so that training can still run in parallel.