Superposition: more features than neurons

When researchers look inside image classifiers, some neurons are easy to name: one fires on curves, another on dog heads. Many are not. In one well-studied vision network, InceptionV1, a single neuron responds to cat faces, the fronts of cars, and cat legs (Olah and colleagues, 2020). In a small language model, Anthropic researchers found a neuron that fires on academic citations, English dialogue, HTTP requests, and Korean text. Neurons like these are called polysemanticPolysemantic neuronA neuron that responds to several unrelated things, such as cat faces and the fronts of cars. Superposition is one proposed explanation for why such neurons are common.Open in glossary, and they make reading a network one neuron at a time much harder.

This lesson trains a model small enough to watch, and shows one reason this could happen.

More features than room

A model has to keep track of many properties of its input: is this text in French, is it a quote, is it about a bridge. Interpretability researchers call these properties featuresFeatureOne measurable property of an example, given to a model as a number, such as a fruit's weight or a pixel's brightness.Taught in Turning things into numbersOpen in glossary, and a common working assumption is that a model represents each feature as a direction in its space of activations.

A layer with $m$ numbers has room for only $m$ directions that do not overlap at all. If a model needs more features than that, it has two choices: drop some, or give several features directions that overlap. Overlap has a cost. When two overlapping features are active together, reading one of them picks up part of the other. That error is called interference.

But most features are rare. Any given sentence is not about bridges, not in Korean, not a citation. If features are seldom active together, overlap seldom costs anything, and storing more features than dimensions can be worth it. Elhage and colleagues at Anthropic (2022) called this superpositionSuperpositionWhen a network represents more features than it has dimensions by giving them overlapping directions. It works when features are rarely active together, at the cost of some interference.Open in glossary and studied it in toy models you can train in seconds.

The toy model

Five features go in, each a number between 0 and 1. The model squeezes them into two numbers, $h$, then tries to rebuild all five:

$W$ is a 2 by 5 matrix, so each feature $i$ gets a direction $W_i$, a column you can draw as an arrow in 2D. $b$ is a bias per feature. Training minimizes the squared error, weighted by how important each feature is:

Each feature is active in some fraction of inputs, its density. When active, its value is random between 0 and 1; otherwise it is 0. Importance falls off from feature 1 to feature 5, $I_i = 0.9^{\,i-1}$ by default.

Watch the features arrange themselves

The model below trains for 3,000 steps of Adam in your browser each time you change a setting. Start with every feature active all the time, then make them rarer.

import numpy as np

def train(density, n=5, m=2, steps=3000, batch=256, lr=0.02, seed=0):
    """Train x' = ReLU(W^T W x + b) on sparse features with Adam."""
    rng = np.random.default_rng(seed)
    W = rng.normal(0, np.sqrt(2 / (n + m)), (m, n))  # column i: feature i's direction
    b = np.zeros(n)
    importance = 0.9 ** np.arange(n)
    params, moments = [W, b], [[np.zeros_like(W), np.zeros_like(W)], [np.zeros_like(b), np.zeros_like(b)]]
    for t in range(1, steps + 1):
        # Each feature is active with probability `density`, then uniform on [0, 1].
        x = rng.random((batch, n)) * (rng.random((batch, n)) < density)
        h = x @ W.T                      # squeeze 5 numbers into 2
        pre = h @ W + b                  # read 5 numbers back out
        out = np.maximum(pre, 0)
        g = 2 * importance * (out - x) * (pre > 0) / (batch * n)  # d loss / d pre
        grads = [h.T @ g + (g @ W.T).T @ x, g.sum(0)]
        step_lr = lr * (1 - (t - 1) / steps)
        for p, grad, (mo, v) in zip(params, grads, moments):
            mo[:] = 0.9 * mo + 0.1 * grad
            v[:] = 0.999 * v + 0.001 * grad**2
            p -= step_lr * (mo / (1 - 0.9**t)) / (np.sqrt(v / (1 - 0.999**t)) + 1e-8)
    return W, b

for density in [1.0, 0.3, 0.05]:
    W, b = train(density)
    lengths = np.linalg.norm(W, axis=0)
    stored = np.flatnonzero(lengths > 0.5)
    angles = np.sort(np.degrees(np.arctan2(W[1], W[0]))[stored])
    gaps = np.diff(np.append(angles, angles[0] + 360))
    print(f"active {density:.0%}: stores features {(stored + 1).tolist()}")
    print(f"  angles between neighbors: {np.round(gaps).astype(int).tolist()}")

active 100%: stores features [1, 2]
  angles between neighbors: [90, 270]
active 30%: stores features [1, 2, 4, 5]
  angles between neighbors: [90, 90, 90, 90]
active 5%: stores features [1, 2, 3, 4, 5]
  angles between neighbors: [70, 71, 73, 73, 72]

What you just saw

Dense features: one dimension each. When every feature is active in every input, overlap would cost something on every example. The model stores the two most important features at right angles and drops the other three, in every seed we tried. That is what principal component analysis would do: keep the directions that matter most.

Rarer features: antipodal pairs. As features get rarer, the model starts storing two features on the same line, pointing in opposite directions. With features active 30% of the time, the default run stores four features as two such pairs (six of the eight seeds we checked did the same). An antipodal pair works because of the ReLU. When one feature of a pair is on, its partner, pointing the other way, reads a negative number, and the ReLU turns it into zero.

Very sparse features: a pentagon. At 5% or 7% density, all five features fit, spaced about 72 degrees apart, in every seed we tried. Each pair now overlaps a little. Look at the biases: they turn negative, so small amounts of interference fall below zero and are cut off.

Importance decides who gets dropped. With steeply falling importance, the least important features are not worth the interference they would cause, and even at 10% density the model stores four, not five.

The ReLU is essential. Switch it off and, with falling or steep importance, the model stores only the two most important features at every density. Without a nonlinearity there is no way to filter interference, so overlapping directions never pay off. With equal importance the linear model may spread its two dimensions across three or more shorter arrows, but that is a projection, not superposition: the stored features per dimension stays at 1.00, while the ReLU model reaches about 3 at 5% density.

The toy shows the trade at the heart of superposition. It buys capacity, and it costs errors whenever features that share space show up together. Switch on two features from a pentagon that sit 144 degrees apart, and the feature between them reads back as present, even though it is off.

Why superposition makes neurons hard to read

In a real network, the numbers inside an MLP layer are neurons, and each passes through its own nonlinearity, so the neuron axes are special. If features have their own directions and those directions line up with neurons, each neuron means one thing. If the model stores more features than it has neurons, the feature directions cannot all line up with neurons. Each neuron then carries a slice of several unrelated features, and it fires on all of them. That is a polysemantic neuron.

Superposition is a hypothesis about why polysemantic neurons are so common, supported by toy models like this one. It also means that looking at neurons one at a time, as you might hope to, can mislead you.

Pulling features back apart

If features are directions that do not line up with neurons, maybe you can find the directions instead. Sparse autoencodersSparse autoencoderA network trained to rebuild a model's activations from a much larger set of features, only a few active at a time. Used to pull apart features stored in superposition; a form of dictionary learning.Open in glossary try to. A sparse autoencoder is a second, separate network trained on a model’s activations. It maps each activation to a much longer list of candidate features, encouraged to keep only a few of them active at once, and then rebuilds the original activation from those few. This is a form of dictionary learning: each candidate feature is a direction in the original space, and every activation is approximately a sparse combination of them.

Bricken and colleagues at Anthropic (2023) applied this to a one-layer transformer whose MLP layer has 512 neurons. They trained dictionaries from 512 up to 131,072 features and studied one with 4,096 in detail. Many of its features were far easier to interpret than the neurons, representing things like DNA sequences, legal language, HTTP requests, and Hebrew text. In blind ratings, people scored the features as more interpretable than the neurons.

Templeton and colleagues (2024) scaled the method up to Claude 3 Sonnet, a production language model, training sparse autoencoders with up to about 34 million features on the activations of a middle layer. They found features for specific concepts, including one for the Golden Gate Bridge. When they forced that feature to be strongly active, the model brought up the bridge in most of its answers, even when it was irrelevant, which is evidence that the feature affects behavior and does not merely correlate with it.

Sparse autoencoders are a tool, not a solution. They rebuild activations imperfectly, so some of what the model computes is left unexplained. There is no guarantee that the features they find are the ones the model uses, or that the list is complete. Naming a feature means reading the examples that activate it, which takes judgment. These are active research questions.