Switching parts off and steering

Reading a model layer by layer showed what is inside SmolLM2-135M at each depth. Reading cannot tell you what the model uses. For that you need an experiment: change one part while the model runs and see whether the output changes. This lesson runs three kinds of experiment on the same model: switching attention heads off, copying activations from one prompt into another, and pushing the activations in a chosen direction.

Layers and heads are numbered from 1, as in the lab. “After layer 24” means the residual stream once 24 layers have run; “after layer 0” is the embedding.

Switching heads off

AblationAblationSwitching off part of a model, for example setting an attention head's output to zero, to see which behaviors depend on it.Taught in How circuits form: induction headsOpen in glossary removes a part and runs the model again. For an attention head, the simplest version replaces its output with zeros. A gentler version replaces it with the head’s average output over the prompt’s positions, so later layers still receive a typical value, just one that says nothing about the particular token.

On The Eiffel Tower is located in the city of, the model gives " Paris" a probability of 88%. Switching off each of the 270 heads in turn, with zeros:

• 237 of the 270 heads change that probability by less than 5 percentage points.
• Layer 2, head 1 has by far the largest effect: without it, " Paris" falls to 16%. With its average instead of zero, it falls to 47%.
• The next largest drops are layer 1, head 5 (to 59%) and layer 24, head 5 (to 61%).
• Some heads work against the answer. Without layer 2, head 6, " Paris" rises to 94%.

Copying activations between prompts

Activation patchingActivation patchingRunning a model on one prompt while copying in an internal activation from a run on another prompt, to test whether that activation carries the information that changes the output.Open in glossary runs the model on two prompts that differ in one detail, here the landmark, then runs one prompt again while copying in part of the other’s residual stream. If copying a piece makes the answer change, that piece carried the information the answer depends on.

Both prompts are 10 tokens long. The subject is split into three tokens in each: " E", "iffel", " Tower" and " Col", "osse", "um". These are the measured probabilities after copying from the Eiffel Tower run into the Colosseum run:

Read the first two rows together. Up to layer 23, everything that makes the answer Paris instead of Rome sits at the subject’s positions: swap them and the answer follows, swap the last position and nothing happens. After layer 24, it is the other way around. Attention is the only part of a transformer that moves information from one position to another, so the move happens in layer 24’s attention heads. Two earlier measurements fit: layer 24, head 5 was among the largest single-head drops above, and in the first lesson it gave the second largest direct push toward " Paris" of all 270 heads. Neither proves that this head does the moving, but together they make it the first place to look.

This style of experiment is often called causal tracingCausal tracingFinding where a model stores and moves information by restoring or swapping activations layer by layer and position by position and measuring the effect on the output.Open in glossary, after Meng, Bau, Andonian, and Belinkov (2022). They corrupted the subject in GPT models with noise and restored one activation at a time, and found that MLP layers in the middle of the network, at the subject’s last token, were decisive, with attention carrying the result to the last position later. The second half of that picture matches what you see here. The third row does not: in this small model and this pair of prompts, the subject’s last token alone was not enough at any layer. All three subject tokens had to be copied together. The setups also differ: in causal tracing the other subject tokens carry noise, while here they still spell out the Colosseum.

Steering with a direction

In the superposition lesson, forcing one sparse autoencoder feature on made a large model bring up the Golden Gate Bridge. A steering vectorSteering vectorA direction added to a model's residual stream while it runs, usually the difference between its average activations on two contrasting sets of prompts, used to push its output one way.Open in glossary gets a direction more cheaply, without training anything:

1. Run the model on a few sentences of one kind (“I love this. It is wonderful.”) and a few of the opposite kind (“I hate this. It is terrible.”).
2. Average each set’s residual stream after one layer, and subtract one average from the other.
3. While the model writes, add a multiple of that difference to the stream after the same layer, at every position.

Turner and colleagues (2023) introduced this as activation addition, with “Love” versus “Hate” as an example prompt pair. They used a single pair of prompts and added the difference at the first few positions of the prompt. The lab instead averages three sentences per side, skips the first position, whose stream reflects only the first word, and adds the result at every position.

Measured with the default settings (after layer 4, three sentences per side), continuing I think my neighbor is and always picking the most likely token (↵ marks a new line):

import numpy as np

# Residual streams (length 4) for prompts on each side, averaged per side.
happy = np.array([[0.9, 0.1, 0.3, 0.2],
                  [0.8, 0.0, 0.4, 0.1]])
angry = np.array([[0.1, 0.9, 0.3, 0.2],
                  [0.0, 0.8, 0.5, 0.1]])
vector = happy.mean(axis=0) - angry.mean(axis=0)
print("steering vector:", np.round(vector, 2))

# Add it, scaled, to a new prompt's residual stream and read the output.
W_U = np.array([[1.0, -1.0, 0.0, 0.0],   # " kind"
                [-1.0, 1.0, 0.0, 0.0]])  # " rude"
h = np.array([0.4, 0.5, 0.3, 0.2])
for strength in (0, 1, -1):
    logits = W_U @ (h + strength * vector)
    p = np.exp(logits - logits.max())
    p /= p.sum()
    print(f"strength {strength:+d}: P(kind) = {p[0]:.2f}")

steering vector: [ 0.8  -0.8  -0.05  0.  ]
strength +0: P(kind) = 0.45
strength +1: P(kind) = 0.95
strength -1: P(kind) = 0.03

A difference of averages over three sentences per side is a blunt tool. It captures everything that differs between the two sets, not only the feeling, and in this 135-million-parameter model the range of strengths where the text changes but still reads as language is narrow. Later methods, such as contrastive activation addition (Panickssery and colleagues, 2024), average over hundreds or thousands of prompt pairs.

What these experiments can and cannot show

• Ablation shows that an output depends on a part, for one prompt. It does not show what the part computes. Zeros are also an unnatural input, which is why the average is often the fairer replacement.
• Backups hide parts. Models often spread a job across several heads, so switching one off can leave the output unchanged even when the job matters. You saw this in How circuits form, where switching off one of several induction heads made copying only somewhat worse, because the others covered for it.
• Patching locates where the difference between two prompts sits and when it moves. It does not explain how the fact was retrieved.
• Steering shows that adding a direction changes behavior. It does not show that the model represents the concept you had in mind as that exact direction.

Each of these is stronger evidence than reading activations alone, because each changes the computation and measures the consequence. None of them, on a few prompts, explains a model.