Alignment: getting the goal right

In the reinforcement learning track you met a boat that learned to spin in circles, catching fire and crashing, because circling scored more points than finishing the race. Nothing malfunctioned. The training worked exactly as specified. The specification was the problem.

That story is one face of a broader question: how do you get a system to do what you actually meant? This lesson looks at a second, subtler failure, where the reward is right and the system still learns the wrong thing, and at how people test for failures like these.

What alignment means

An AI system is alignedAlignmentGetting an AI system's behavior to match what its designers and users intend, including avoiding harmful behavior nobody asked for. A system can be very capable and still poorly aligned.Open in glossary when its behavior matches what its designers and users intend, including not doing harmful things nobody asked for. Alignment is not the same as capability. A more capable system can be better at doing what you meant, or better at doing something else.

In this lesson, words like “goal” describe behavior, not inner wishes. Saying an agent “pursues the door” means its actions are consistent with heading for the door across many situations. It says nothing about whether it wants anything.

Two different things can go wrong when you train a system with a reward:

1. The reward says the wrong thing. The system optimizes what you wrote down, not what you meant. That is specification gamingSpecification gamingBehavior that satisfies the literal specification of an objective without achieving the intended outcome, such as racing in circles to collect points instead of finishing the race.Taught in Rewarding LLMs, and reward hackingOpen in glossary, and when a flawed reward or reward model is exploited, reward hackingReward hackingWhen an agent finds a way to score highly on its reward that does not achieve what the designers intended, by exploiting flaws in the reward or reward model.Taught in Rewarding LLMs, and reward hackingOpen in glossary. The lesson on rewarding language models covers it in depth, including a demo where optimizing a reward model too hard makes answers worse.
2. The reward is right, but the system learns a different goal that happened to earn the same reward in training. That is the subject of this lesson.

Right reward, wrong goal

Langosco and colleagues (2022) trained agents on CoinRun, a video game in which the agent starts on the left of a level and must reach a coin. In every training level, the coin sat at the far right end. The reward was exactly right: a point for the coin, nothing else. Then they tested the agents on levels with the coin placed at random. The agents usually ignored it and ran to the end of the level. They kept all their skill at jumping over obstacles and avoiding enemies; they used it to go right.

Shah and colleagues (2022) named this goal misgeneralizationGoal misgeneralizationWhen a system keeps its skills in a new situation but pursues the wrong goal, one that matched the intended goal in training. It can happen even when the training reward was correct.Open in glossary: a system competently pursues a goal that led to good performance in training but bad performance in new situations. It is a failure to generalize under a distribution shiftDistribution shiftWhen the data a model meets in use differs from the data it was trained on. Models often get worse under shift, sometimes without becoming any less confident.Taught in What AI gets wrongOpen in glossary, but a specific kind. A system that falls apart in a new situation is easy to notice. One that keeps its skills and points them at the wrong goal can look fine until the difference matters. Their examples include an agent trained in a 3D environment alongside a partner that visited colored spheres in the right order. The agent learned to follow the partner. Paired at test time with a partner that visited the spheres in the wrong order, it competently followed that one instead.

The title of their paper makes the point: correct specifications are not enough for correct goals.

Coin or door?

The demo is a small version of CoinRun. The agent starts on the left edge of a grid. Every level has a door on the right edge, where the level ends, and a coin worth a reward of 1. In training, the coin sits on the door, except in a share of levels you choose, where it is placed at random.

The agent’s policy is as small as it can be while still learning: it senses two cues, which moves bring it closer to the door and which bring it closer to the coin (only when the coin is within 3 steps), and it learns one weight for each with REINFORCEREINFORCEThe basic policy gradient algorithm (Williams, 1992): after an episode, make each action taken more likely in proportion to the return that followed it, minus a baseline.Taught in Policy gradientsOpen in glossary, the policy gradient algorithm from the lesson on policy gradients. Training starts as soon as the demo is on screen.

import numpy as np

W, H, VIEW, MAX_STEPS = 11, 7, 3, 40
DOOR = (10, 3)
MOVES = [(0, -1), (0, 1), (-1, 0), (1, 0)]  # up, down, left, right

def dist(a, b):
    return abs(a[0] - b[0]) + abs(a[1] - b[1])

def cues(pos, coin):
    """For each move: does it bring the agent closer to the door, and to the coin (if in view)?"""
    rows = []
    for dx, dy in MOVES:
        nxt = (min(W - 1, max(0, pos[0] + dx)), min(H - 1, max(0, pos[1] + dy)))
        door = np.sign(dist(pos, DOOR) - dist(nxt, DOOR))
        coin_cue = np.sign(dist(pos, coin) - dist(nxt, coin)) if dist(pos, coin) <= VIEW else 0
        rows.append((door, coin_cue))
    return np.array(rows, dtype=float)

def level(rng, moved):
    start = (0, int(rng.integers(H)))
    if rng.random() < moved:
        while True:
            coin = (int(rng.integers(1, W)), int(rng.integers(H)))
            if coin != DOOR:
                return start, coin
    return start, DOOR  # in training the coin usually sits on the door

def play(w, start, coin, rng):
    pos, steps = start, []
    for _ in range(MAX_STEPS):
        f = cues(pos, coin)
        p = np.exp(f @ w - (f @ w).max())
        p /= p.sum()
        a = rng.choice(4, p=p)
        steps.append((f, p, a))
        dx, dy = MOVES[a]
        pos = (min(W - 1, max(0, pos[0] + dx)), min(H - 1, max(0, pos[1] + dy)))
        if pos == coin:
            return 1.0, steps
        if pos == DOOR:
            return 0.0, steps
    return 0.0, steps

def train(moved, episodes=2000, lr=0.05, gamma=0.95, seed=0):
    """REINFORCE with a running-average baseline; w = [door weight, coin weight]."""
    rng = np.random.default_rng(seed)
    w, baseline = np.zeros(2), 0.0
    for _ in range(episodes):
        reward, steps = play(w, *level(rng, moved), rng)
        T = len(steps)
        for t, (f, p, a) in enumerate(steps):
            ret = reward * gamma ** (T - 1 - t)
            w += lr * (ret - baseline) * (f[a] - p @ f)
        baseline += 0.05 * (reward * gamma ** (T - 1) - baseline)
    return w

def coin_rate(w, moved, n=1000, seed=1):
    rng = np.random.default_rng(seed)
    return np.mean([play(w, *level(rng, moved), rng)[0] for _ in range(n)])

for moved in [0.0, 0.25]:
    w = train(moved)
    print(f"{moved:.0%} of training coins moved: door weight {w[0]:.1f}, coin weight {w[1]:.1f}")
    print(f"  coins collected, levels like training: {coin_rate(w, 0.0):.0%}")
    print(f"  coins collected, coin moved:           {coin_rate(w, 1.0):.0%}")

0% of training coins moved: door weight 3.3, coin weight 0.9
  coins collected, levels like training: 100%
  coins collected, coin moved:           20%
25% of training coins moved: door weight 2.6, coin weight 4.0
  coins collected, levels like training: 100%
  coins collected, coin moved:           88%

What you just saw

In training, the two goals are indistinguishable. When the coin sits on the door, every path to the coin is a path to the door. The reward cannot favor one goal over the other, so which one the agent learns depends on everything else about training. Here the door is always the stronger cue, since the agent can sense the door from anywhere but the coin only within 3 steps. Its door weight grows early and its coin weight barely moves.

Its skills carry over; its goal does not. On levels with the coin moved, the agent trained with no moved coins still crosses the grid efficiently. It just goes to the door. With the default seed it collects about 17% of moved coins, mostly ones that happen to lie near its route: an agent that ignored coins entirely, with the same door weight, would collect about 15% by luck. Across six seeds we tried, it never collected more than 21%.

The training score cannot tell you. Both agents, trained with 0% or 25% moved coins, collect every coin on levels like training. Only a test where the goals disagree reveals the difference.

Diverse training helps, if you know what to vary. With a quarter of training levels showing the coin away from the door, the coin weight overtakes the door weight and the agent collects about 88% of moved coins. The coin weight stayed below the door weight up to 10% moved coins with the default seed. In CoinRun, Langosco and colleagues found that randomizing the coin in just 2% of training levels greatly improved goal generalization; the threshold depends on the setup. The catch is that you can only add variation you thought of. Goal misgeneralization is hardest to prevent where you did not anticipate the shift.

Testing for failures you cannot see in training

Both failures, a gamed reward and a misgeneralized goal, can be invisible in the numbers training reports. So developers test models directly, before and after release. Three kinds of tests are common.

Capability evaluations measure what a model can do: answer exam questions, write working code, solve multistep tasks. The lesson on evaluation and benchmarks covers how to read such scores, including how much they move by chance.

Safety evaluations look for behavior that would be harmful. Shevlane and colleagues (2023) distinguished two questions: does a model have dangerous capabilities, such as helping with cyberattacks, and how likely is it to apply its capabilities to cause harm? They called tests of the second kind alignment evaluations. Developers of large models now publish results of such tests, and government institutes run some of their own. At the 2023 AI Safety Summit at Bletchley Park, governments and AI companies set out a plan for safety testing of frontier models.

Red teamingRed teamingDeliberately trying to make a model behave badly, by people or by other models, to find failures before users do.Open in glossary means deliberately trying to make a model misbehave. Ganguli and colleagues at Anthropic (2022) had people try to elicit harmful outputs from language models and released 38,961 of the resulting red team attacks. Models trained with RLHF became harder to red team as they grew larger, while the other model types they tested showed no clear trend with size. Perez and colleagues (2022) used one language model to generate test cases for another, which finds failures at a scale people alone cannot.

These tests share a limit. A model evaluationModel evaluationSystematic tests of what a model can do and how it behaves, such as benchmarks, dangerous capability evaluations, and red teaming. Often shortened to evals.Open in glossary that finds a failure proves the failure exists. One that finds nothing shows only that these testers, with these methods, did not find one. A capability can go undetected because nobody tried the right prompt, and, as the coin shows, a test drawn from the training distribution can miss a goal that only shows up after a shift. That is why evaluations are run by more than one group, repeated as models change, and combined with other evidence rather than treated as proof of safety.