How circuits form: induction heads

Mention a name early in a prompt, and a language model is far more likely to use it again later. Show it two examples of a format, and it follows the format. Nothing in its training told it about your prompt. It picks the pattern up from the text in front of it, with its weights unchanged. This is called in-context learningIn-context learningA model picking up a pattern from the text in its prompt, such as repeating a name or following an example, without any change to its weights.Open in glossary.

One of the simplest mechanisms behind it was found by taking small transformers apart, and you can watch it form. In this lesson you will train a tiny transformer from scratch, in your browser, and see a two-part circuitCircuitA small part of a neural network, such as a few attention heads in different layers, that together carry out one specific, understandable computation.Open in glossary appear inside it, suddenly, partway through training.

Copying takes two steps

Suppose the text so far is Q J F K R B Q J F. A good guess for the next letter is K: the last time F appeared, K came next. To make that guess, the model has to find the earlier F and read what followed it.

Attention cannot do that in one step. An attention head compares the current position’s query with each earlier position’s key. The current letter is F, so its query can look for keys that say “F”. But the position we want is the one holding K, and nothing about K itself says “F”. What makes that position special is the letter before it.

So the model splits the job across two layers:

1. A previous-token headPrevious-token headAn attention head that mostly looks at the position just before the current one, writing information about the previous token into each position.Open in glossary in layer 1. At every position, it looks one step back and writes “the letter before me was …” into that position. Now the position holding K also carries “the letter before me was F”.
2. An induction headInduction headAn attention head that, at a token A, looks back for earlier copies of A and attends to the token that followed them, helping the model continue repeated patterns.Taught in Many heads and the causal maskOpen in glossary in layer 2. At the current F, it searches for positions whose previous letter was F, finds the one holding K, and copies K into the prediction.

Elhage and colleagues described this two-head mechanism in 2021, studying transformers made of attention layers only, and named the second head an induction head. Layer 2’s head can only search on “the letter before me” because layer 1 put that information there first: the two heads are composed.

Watch it form

The model below is a two-layer transformer with four attention heads per layer and nothing else between its input and output layers: no MLPs and no normalization. That is about 11,000 numbers in all. It trains on sequences of letters from A to X. Each sequence starts with a random stretch of letters, then repeats that stretch exactly, then continues with more random letters. The length of the repeated stretch changes from sequence to sequence, so the model cannot copy from a fixed distance back; it has to find the earlier occurrence by content.

The random letters cannot be predicted, so the only thing to learn is copying. The blue line tracks the loss on letters inside the repeat, which copying can predict. The dashed orange line tracks every other letter.

import numpy as np

# A start marker, the letters Q J F K, then the same four again.
tokens = ["start", "Q", "J", "F", "K", "Q", "J", "F", "K"]
T = len(tokens)
first, second, length = 1, 5, 4  # the copy begins at position 5
gap = second - first

def causal_softmax(scores):
    """Turn scores into attention: each position sees itself and earlier ones."""
    masked = np.where(np.tril(np.ones((T, T))) == 1, scores, -np.inf)
    w = np.exp(masked - masked.max(axis=1, keepdims=True))
    return w / w.sum(axis=1, keepdims=True)

# A layer-1 head that strongly prefers the position just before it.
prev_scores = np.zeros((T, T))
for i in range(1, T):
    prev_scores[i, i - 1] = 4.0

# A layer-2 head that, inside the copy, prefers the letter after the
# earlier occurrence of the current letter.
ind_scores = np.zeros((T, T))
for i in range(second, T - 1):
    ind_scores[i, i - gap + 1] = 2.0

prev_attn = causal_softmax(prev_scores)
ind_attn = causal_softmax(ind_scores)

# The demo's scores: average over queries inside the copy (but its last letter).
queries = range(second, second + length - 1)
prev_score = np.mean([prev_attn[i, i - 1] for i in queries])
ind_score = np.mean([ind_attn[i, i - gap + 1] for i in queries])
print(f"previous-letter score: {prev_score:.2f}")
print(f"induction score: {ind_score:.2f}")
print("letter F at position 7 looks most at:", tokens[int(ind_attn[7].argmax())], "at position", int(ind_attn[7].argmax()))

previous-letter score: 0.90
induction score: 0.55
letter F at position 7 looks most at: K at position 4

What you just watched

A plateau, then a phase changePhase changeA sudden change in a model's behavior or training loss over a short stretch of training, often because a new mechanism has formed inside the model.Open in glossary. For the first few hundred steps the loss on copied letters barely moves. Then, over a short stretch, it falls far below guessing. Nothing about the training changed at that moment: same data, same learning rate. What changed was inside the model. A common way to think about why the change is sudden: the two heads only help together. A previous-letter head writes information that nothing reads yet, and an induction head has nothing to match on without it, so each part gets little reward until the other starts to form. Once both exist, they reinforce each other quickly.

The scores move with the loss. The induction score measures how much attention a letter in the copy pays to the letter after its earlier occurrence. It jumps during the same steps in which the loss falls, which is evidence that this head pattern is what does the copying. The previous-letter head forms more gradually. In one of our longer runs its score was about 0.2 at the phase change and close to 0.9 by step 5,000.

Random letters stay random. The loss on letters outside the repeat ends near ln 24, the cost of guessing among 24 letters, and even a little above it, because the model sometimes bets on a copy that is not there.

Testing the circuit by switching heads off

Watching attention patterns suggests a story; ablationAblationSwitching off part of a model, for example setting an attention head's output to zero, to see which behaviors depend on it.Open in glossary tests it. Set one head’s output to zero and measure what breaks. In our test runs on this model:

• Switching off any single layer-2 head made copying worse, but by less than a factor of two in the loss. Several layer-2 heads became induction heads, so the others cover for the missing one.
• Switching off the previous-letter head in layer 1, after about 1,200 steps of training, multiplied the loss on copied letters by roughly two to five. All four induction heads were untouched, but each of them depended on what that one head writes.

A behavior can survive the loss of any one of several redundant parts and still collapse when a single shared input disappears. That is why interpretability researchers test mechanisms by intervening, not only by looking.

What is known about real models

Olsson and colleagues (2022) followed this mechanism into much larger models. They found induction heads in a wide range of transformers, including large language models. In their models the heads formed during a phase change early in training, visible as a bump in the training loss, at the same point where in-context learning sharply improved. They presented several lines of evidence that induction heads could account for most in-context learning in small attention-only models. For large models with many more components, their evidence was weaker and mostly correlational.

This toy makes the circuit unusually clean. The model is tiny, it is trained on nothing but copying, and its sequences are designed so copying is the only thing to learn. A real language model learns this circuit alongside everything else, and also develops looser versions that complete patterns of similar, not just identical, tokens.