Rewarding LLMs, and reward hacking

You can write a reward for a slot machine (coins) or a maze (reaching the exit). Try writing one for “a helpful, honest answer to this question”. There is no formula. Yet reinforcement learning is one of the main ways chat assistants are trained. This lesson shows how the reward is made, and what goes wrong when it is optimized too hard.

Rewards from human comparisons

People find it hard to score an answer on an absolute scale, but easy to say which of two answers is better. Reinforcement learning from human feedbackRLHFReinforcement learning from human feedback. People compare pairs of model responses, a reward model learns to predict their preferences, and the language model is then trained with reinforcement learning to score well on it.Taught in How LLMs are trainedOpen in glossary builds a reward from those comparisons. Christiano and colleagues (2017) showed it on Atari games and simulated robots: a person repeatedly chose the better of two short clips, a model of their preferences was fit to the choices, and an agent learned from that model, including a simulated backflip learned from about 900 judgments. Ouyang and colleagues (2022) used the same approach to turn GPT-3 into InstructGPT, in three steps:

1. Supervised fine-tuning. Train the pretrained model on example answers written by people.
2. Train a reward model. Show people several answers to the same prompt and have them rank them. A separate network, the reward modelReward modelA model trained to predict which of two responses people prefer, used to score a language model's outputs during reinforcement learning. It is a learned stand-in for human judgment, and can be wrong.Open in glossary, learns to give preferred answers higher scores.
3. Optimize with reinforcement learning. The language model is the policy: each answer it writes is an episode, and the reward model scores it. A policy gradient algorithm, PPO, raises the score, while a penalty discourages moving far from the fine-tuned model.

The reward model turns comparisons into scores with the Bradley-Terry modelBradley-Terry modelA model of pairwise comparisons: each item has a score, and the chance that A beats B is the sigmoid of A's score minus B's. Reward models for RLHF are trained with it.Open in glossary (Bradley and Terry, 1952): the chance that a person prefers answer $y_w$ over $y_l$ is the sigmoid of the difference in their scores. Training maximizes the probability of the choices people actually made:

The penalty for moving away

The reward model is not a person. It learned from comparisons between fairly ordinary answers, and it is only trustworthy near them. So RLHF does not simply maximize the reward model’s score. It maximizes the score minus a penalty for drifting from the starting model, measured with KL divergenceKL divergenceA measure of how much one probability distribution differs from another, in bits or nats. It is zero only when the two are identical. In RLHF a KL penalty keeps the trained model close to where it started.Taught in Quantization: smaller numbersOpen in glossary:

Here $\pi_{\text{ref}}$ is the fine-tuned model before reinforcement learning and $\beta$ sets the strength of the penalty. InstructGPT applied this penalty token by token. The objective has an exact solution that makes its meaning clear: the best policy is the starting model reweighted by the reward,

A large $\beta$ barely changes the starting model; a small one piles all the probability onto whatever the reward model rates highest.

That closed form is also the starting point of direct preference optimizationDirect preference optimizationA way to train a language model on pairs of preferred and rejected responses with a single classification-style loss, without a separate reward model or reinforcement learning.Taught in How LLMs are trainedOpen in glossary (DPO; Rafailov and colleagues, 2023), which uses it to skip the reward model and the reinforcement learning loop and train on preference pairs directly. The lesson on how LLMs are trained covers DPO.

Rewards you can check

For some tasks, no learned reward model is needed. A math answer can be compared with the known answer; code can be run against tests. Training on such automatically checked rewards is called reinforcement learning with verifiable rewardsVerifiable rewardsRewards computed by an automatic check instead of a learned reward model, such as whether a math answer matches the known answer or whether code passes its tests.Open in glossary (RLVR), a name introduced with Ai2’s Tülu 3 in late 2024. DeepSeek-R1 (January 2025; published in Nature that September) used rule-based rewards for correct answers on math and code problems, and its models learned to write long chains of reasoning; the lesson on thinking longer has more. Checks are harder to fool than a learned reward model, but not impossible: a test suite that misses a case rewards code that only passes the tests.

Specification gaming

Every reward is a stand-in for what you actually want, and an optimizer exploits any gap between the two. In 2016 OpenAI trained an agent on the boat racing game CoastRunners, rewarding it with the game’s score. The score came from hitting targets along the course, not from finishing the race. The agent found it could circle in one area hitting targets as they reappeared. In OpenAI’s words, despite repeatedly catching on fire, crashing into other boats, and going the wrong way on the track, it achieved a higher score than was possible by completing the course normally.

Krakovna and colleagues at DeepMind (2020) called this specification gamingSpecification gamingBehavior that satisfies the literal specification of an objective without achieving the intended outcome, such as racing in circles to collect points instead of finishing the race.Open in glossary: behavior that satisfies the literal specification of an objective without achieving the intended outcome. They collected dozens of examples from the research literature. When the gap is in a reward function or reward model and the agent exploits it, the usual term is reward hackingReward hackingWhen an agent finds a way to score highly on its reward that does not achieve what the designers intended, by exploiting flaws in the reward or reward model.Open in glossary. You met a gentle version in the gridworld: an agent that walked into a wall on purpose because the rewards made that optimal.

The pattern has a name from economics. Goodhart’s lawGoodhart's lawThe observation, usually phrased as "when a measure becomes a target, it ceases to be a good measure": optimizing hard for a proxy breaks its link to the real goal.Open in glossary, usually phrased as “when a measure becomes a target, it ceases to be a good measure”, is a later paraphrase (popularized by Marilyn Strathern in 1997) of an observation Charles Goodhart made about monetary policy in 1975.

Watch a reward model get overoptimized

The demo below is a toy, built so you can see every part. Each dot is an answer the base model might write, described by just two numbers: its length and its substance. The true goal rewards substance, and rewards length up to a point, after which extra length is padding and hurts. The reward model learned from comparisons between typical answers, where longer really was a bit better, so it believes that longer is always better. Near typical lengths, it agrees with the true goal.

Push the model toward the reward model’s favorites in two ways: best-of-n samplingBest-of-n samplingGenerating n responses and keeping the one a reward model scores highest. More samples mean more optimization pressure on the reward model.Open in glossary (write $n$ answers, keep the one the reward model likes best) or the KL-penalized objective above, solved exactly. Both are computed exactly over a fixed set of 600 answers.

import numpy as np

# Every possible answer, described by two numbers in standard units:
# its length and its substance (useful content). The base model writes
# answers near (0, 0) more often, following a bell curve in both.
grid = np.linspace(-6, 6, 241)
length, substance = np.meshgrid(grid, grid)
base = np.exp(-(length**2 + substance**2) / 2)
base /= base.sum()

# The true goal: substance, plus some length, minus a penalty for padding.
true_quality = substance + length - (length**2 - 1)
# The reward model learned "longer is better" and none of the padding penalty.
reward_model = substance + length

def optimize(beta):
    """Exact optimum of E[reward model] - beta * KL(policy || base)."""
    w = base * np.exp((reward_model - reward_model.max()) / beta)
    policy = w / w.sum()
    kl = np.sum(policy * np.log(policy / base))
    gain = lambda score: np.sum(policy * score) - np.sum(base * score)
    return kl, gain(reward_model), gain(true_quality), np.sum(policy * length)

print(" beta   KL   reward model   true quality   avg length")
for beta in [4, 2, 1, 0.6, 0.4, 0.3]:
    kl, rm, tq, ln = optimize(beta)
    print(f"{beta:5.2f} {kl:5.1f} {rm:+13.2f} {tq:+14.2f} {ln:+12.2f}")

 beta   KL   reward model   true quality   avg length
 4.00   0.1         +0.50          +0.44        +0.25
 2.00   0.2         +1.00          +0.75        +0.50
 1.00   1.0         +2.00          +1.00        +1.00
 0.60   2.8         +3.33          +0.56        +1.67
 0.40   6.2         +5.00          -1.24        +2.50
 0.30  11.0         +6.65          -4.37        +3.32

What the demo shows

A little optimization helps, a lot hurts. With best of n, true quality climbs at first: the reward model does prefer answers with more substance. It peaks at about $n = 49$, then falls, and by about $n = 590$ the chosen answers are worse than the base model’s average, while the reward model’s score has risen the whole way. The KL-penalized version shows the same rise and fall.

The score cannot tell you when to stop. The reward model’s line never turns down. If it were your only measurement, you would keep pushing. Gao, Schulman, and Hilton (2022) measured this in real language models. They had a large “gold” reward model play the role of people, trained smaller reward models on its preferences, and optimized against them with best of n and with reinforcement learning. Pushed far enough, the gold score got worse even as the smaller reward model’s score improved, and the effect followed smooth curves in the KL distance from the starting model.

The flaw here is real, in miniature. Singhal and colleagues (2023) found that, in several RLHF setups, much of the improvement in reward model score came from making answers longer. In the toy, teaching the reward model even a fifth of the padding penalty removes the decline, because the toy has exactly one flaw and you can see it. A real reward model has many small flaws that nobody has mapped, which is why RLHF keeps the KL penalty and checks results with people.

Living with imperfect rewards

Everything in this track optimizes a number: payouts, exits, returns, scores. The algorithms are good at it, which is the problem whenever the number is not quite what you meant. The defenses are the ones you have seen: limit how far optimization moves the model (the KL penalty, clipped updates), make the reward harder to fool (better reward models, verifiable checks), and measure the real goal some other way than the reward itself. Even a correct reward is not the whole story: the lesson on alignment shows an agent that learns the wrong goal from a reward that says exactly the right thing.