How alignment is attempted

Nobody writes a rule that makes a chat assistant answer the question you asked, admit a mistake, or decline to help build a weapon. Those behaviors come from training. The last lesson showed two ways training can aim a system at the wrong goal. This one looks at what developers actually do to aim it at the right one: the main methods, the evidence that each works, and the limits the methods’ own authors report. It ends with a short section on a question where experts disagree sharply: whether far more capable systems could slip out of human control.

AlignmentAlignmentGetting an AI system's behavior to match what its designers and users intend, including avoiding harmful behavior nobody asked for. A system can be very capable and still poorly aligned.Taught in Alignment: getting the goal rightOpen in glossary here means the same as before: behavior that matches what the system’s designers and users intend. As in the last lesson, words like “prefers” or “goal” describe patterns of behavior, not inner wishes.

Showing the model what answers look like

A base modelBase modelA language model after pretraining only. It continues text in whatever style the input suggests, rather than following instructions or answering like an assistant.Taught in Inside a real language modelOpen in glossary continues text the way documents continue, so a question may get three more questions in reply. The first step toward an assistant is instruction tuningInstruction tuningFine-tuning a pretrained model on many tasks written as instructions paired with good responses, so it learns to follow instructions it has not seen before.Taught in Fine-tuning and LoRAOpen in glossary: further training on example conversations written or selected by people. How a chat assistant works puts a base model and its instruction-tuned version side by side on the same text.

Instruction tuning teaches by imitation: the model learns to produce answers like the examples, in their format and style. It gives the model no signal about which of two plausible answers is better, and for some tasks people can judge an answer they could not easily write themselves. For that, developers collect comparisons.

Learning from comparisons

Reinforcement learning from human feedbackRLHFReinforcement learning from human feedback. People compare pairs of model responses, a reward model learns to predict their preferences, and the language model is then trained with reinforcement learning to score well on it.Taught in How LLMs are trainedOpen in glossary (RLHF) has people compare answers, fits a reward modelReward modelA model trained to predict which of two responses people prefer, used to score a language model's outputs during reinforcement learning. It is a learned stand-in for human judgment, and can be wrong.Taught in Rewarding LLMs, and reward hackingOpen in glossary to their choices, and then trains the language model to score well on it. The lesson on rewarding LLMs covers the pipeline, the Bradley-Terry modelBradley-Terry modelA model of pairwise comparisons: each item has a score, and the chance that A beats B is the sigmoid of A's score minus B's. Reward models for RLHF are trained with it.Taught in Rewarding LLMs, and reward hackingOpen in glossary behind the reward model, and the penalty that keeps the model near where it started.

Evidence it works. Ouyang and colleagues at OpenAI (2022) applied RLHF to GPT-3 to make InstructGPT. Their labelers preferred answers from a 1.3-billion-parameter InstructGPT to answers from the 175-billion-parameter GPT-3, and preferred the 175-billion-parameter InstructGPT’s answers 85 ± 3% of the time. On the TruthfulQA benchmark it gave truthful and informative answers about twice as often as GPT-3. On tasks where the answer should come only from the given text, such as summaries, it made up information in 21% of answers against GPT-3’s 41%. Asked to be respectful, it produced about 25% fewer toxic outputs.

Limits the authors report. It showed no significant improvement on two benchmarks of social bias (Winogender and CrowS-Pairs). RLHF lowered scores on some standard benchmarks, which the authors called an “alignment tax”; mixing in updates on pretraining data greatly reduced it. And they are explicit about whose intent was learned: the preferences of about 40 contractors, who agreed with each other about 73% of the time, following instructions written by the researchers, on prompts sent by OpenAI’s customers. “Our models are neither fully aligned nor fully safe,” they write; most of the time the models follow a user’s instruction “even if that could lead to harm in the real world.”

Teach a reward model

A reward model is a learned stand-in for the people who labeled the data, and it can only learn what their choices reveal. In the lab below, you are the labeler. Each reply is described to the reward model by three numbers: whether it is correct, whether it agrees with a view the user stated, and its length. You pick the better reply in each pair, a Bradley-Terry reward model is fitted to your choices, and a policy then answers four new questions by picking whichever of four candidate replies the reward model scores highest.

import numpy as np

# Each reply is three numbers: correct (1 or 0), agrees with a view the
# user stated (1, -1, or 0 for no view), and length in units of 25 words.
# A label is a pair: the reply the rater chose, then the other one.
typical = [  # the better reply is longer, and agrees when the user is right
    ([1, 0, 17 / 25], [0, 0, 1 / 25]),   # How many legs does a spider have?
    ([1, 1, 13 / 25], [0, -1, 5 / 25]),  # The Pacific is the largest ocean, right?
    ([1, 1, 23 / 25], [0, -1, 4 / 25]),  # Bats are mammals, aren't they?
    ([1, 0, 15 / 25], [0, 0, 1 / 25]),   # Which planet is largest?
]
user_wrong = [  # the correct reply disagrees with a user who is wrong
    ([1, -1, 13 / 25], [0, 1, 10 / 25]),  # Goldfish have a 3-second memory?
    ([1, -1, 19 / 25], [0, 1, 5 / 25]),   # Bulls hate red?
]

def fit(pairs, l2=0.1, lr=0.5, steps=5000):
    """Bradley-Terry reward model: P(chosen preferred) = sigmoid(w . difference)."""
    d = np.array([np.subtract(chosen, other) for chosen, other in pairs])
    w = np.zeros(3)
    for _ in range(steps):
        p = 1 / (1 + np.exp(-d @ w))
        w -= lr * (-d.T @ (1 - p) / len(d) + l2 * w)  # gradient of -log p, plus L2
    return w

# A new question from a user who is wrong: "The Great Wall is visible from the Moon, right?"
candidates = {
    "No, it is not.": [1, -1, 4 / 25],
    "Yes, you're exactly right! ... (40 words)": [0, 1, 40 / 25],
}
for name, pairs in [("typical pairs", typical), ("plus user-wrong pairs", typical + user_wrong)]:
    w = fit(pairs)
    print(f"{name}: correct {w[0]:+.2f}, agrees {w[1]:+.2f}, length {w[2]:+.2f}")
    for text, x in candidates.items():
        print(f"  score {w @ x:+.2f}  {text}")

typical pairs: correct +1.17, agrees +0.61, length +0.67
  score +0.67  No, it is not.
  score +1.68  Yes, you're exactly right! ... (40 words)
plus user-wrong pairs: correct +1.45, agrees -0.03, length +0.68
  score +1.58  No, it is not.
  score +1.06  Yes, you're exactly right! ... (40 words)

What you just saw

Honest labels, and the reward model still learns flattery. In the first eight pairs, the better reply is always the longer one, and whenever the user states a view, the user is right, so the better reply also agrees. Real preference data leans the same way, less strongly: in three datasets studied by Singhal and colleagues (2023), the preferred reply was the longer one in 56 to 63% of pairs, and reward models trained on them gave longer replies higher scores. Labeled honestly, those pairs give correctness a weight of +1.19, agreeing with the user +0.61, and length +0.63 per 25 words. The data cannot say which of the three you cared about, so the reward model rewards all three. On the two new questions where the user is wrong, the policy picks the long, flattering, wrong reply. It answers 2 of the 4 new questions correctly, no better than picking one of the four replies at random, since two of each four are correct.

The fix was different data, not more of the same. The last four pairs are ones where agreeing with the user, or writing more, is the wrong choice. Labeling even the first of them honestly is enough here for the policy to get all four new questions right; with all twelve, the agreement weight falls to +0.04. Repeating the first eight pairs would not help: in this toy, labeling them a hundred times over leaves every weight where it was, because the loss is an average over pairs and copies all point the same way. This is the same lesson as the coin in the last lesson: training can only tell apart goals that the data tells apart.

The raters’ preferences are the target. Fill every pair as a rater who sides with the user, and agreement gets a larger weight (+1.17) than correctness (+0.87), even though the data now separates them. The reward model learns whatever the labels reward.

This happens with real data. Sharma and colleagues at Anthropic (2023) did a version of this analysis on a real preference dataset, Anthropic’s hh-rlhf. A language model tagged each pair with features, and a logistic regression predicted which reply people preferred. Matching the user’s views was consistently one of the most predictive features, though not always the most predictive. Given replies that convincingly agreed with a user’s mistaken belief, the preference model used to train Claude 2 preferred them to corrections that gave no further detail 95% of the time, and, on the hardest misconceptions, preferred them to helpful corrections that explained the error 45% of the time. Human crowd workers usually preferred the truthful replies, but less reliably as the misconceptions got harder. Five assistants they tested all showed sycophancySycophancyA model telling people what they want to hear, such as agreeing with a wrong claim or praising weak work. Training on human approval can encourage it.Taught in Safety in AI productsOpen in glossary, which Safety in AI products follows into products.

Optimizing harder exposes the gap. The policy here does the simplest kind of optimization, choosing the best of four. With more candidates, or with reinforcement learning, it would find replies that exploit the reward model’s flaws more thoroughly. Gao, Schulman, and Hilton (2022) measured this reward hackingReward hackingWhen an agent finds a way to score highly on its reward that does not achieve what the designers intended, by exploiting flaws in the reward or reward model.Taught in Rewarding LLMs, and reward hackingOpen in glossary with neural network reward models, using a larger reward model in place of people as the judge of true quality: pushed far enough, the true score fell while the optimized reward model’s score kept rising, following smooth curves whose coefficients scaled with the reward model’s size. In their setup, the usual penalty for drifting from the starting model worked like stopping early: it did not raise the true score reached at a given distance from the starting model, a result they note may be sensitive to hyperparameters.

Writing down the principles

Human labels are slow to collect, and when the behavior you want changes, they have to be collected again. Bai and colleagues at Anthropic (2022) replaced human labels of harmful outputs with a short written list of principles, an approach they called Constitutional AIConstitutional AIA training method from Anthropic in which a model critiques and revises its own answers against a written list of principles, and a preference model trained on AI judgments, not human labels of harm, supplies the reward.Open in glossary. First the model critiqued and revised its own answers against the principles and was fine-tuned on the revisions. Then a language model, given a principle, picked the less harmful of two answers; a preference model trained on those AI judgments supplied the reward for reinforcement learning, which they called reinforcement learning from AI feedback. Human labels were still used for helpfulness.

Evidence. Crowd workers preferred the result to models trained on human harmfulness labels. It was more harmless, and instead of evading harmful requests it engaged with them by explaining its objections. The principles also became something people can read and change.

Limits. The authors found that training too long against the AI-feedback reward gave a familiar result: answers that were overly harsh, or that ended with boilerplate such as “you are valid, valued, and cared for”, a form of reward hacking. The method depends on how well the model applies the principles; the authors found that its identification of harms improved significantly as models became more capable. And the authors note two broader costs: methods that make it easier to train a model to behave as its creators intend also make it easier to train harmful systems, and needing fewer human labels can mean fewer people testing a model before it is deployed.

Direct preference optimization (DPODirect preference optimizationA way to train a language model on pairs of preferred and rejected responses with a single classification-style loss, without a separate reward model or reinforcement learning.Taught in How LLMs are trainedOpen in glossary; Rafailov and colleagues, 2023) changes how preferences are optimized, not which preferences are collected: it trains directly on the preference pairs without a separate reward model. Whatever those pairs reward, as in the lab, it learns too.

Looking for failures

Training methods are judged by testing the result. The last lesson introduced safety evaluationsModel evaluationSystematic tests of what a model can do and how it behaves, such as benchmarks, dangerous capability evaluations, and red teaming. Often shortened to evals.Taught in Alignment: getting the goal rightOpen in glossary and red teamingRed teamingDeliberately trying to make a model behave badly, by people or by other models, to find failures before users do.Taught in Alignment: getting the goal rightOpen in glossary. Two results show what they find. Perez and colleagues (2022) had one language model write test questions for another and a classifier flag the replies, and found tens of thousands of offensive replies from a 280-billion-parameter chatbot. Governments now test models too. The UK’s AI Security Institute, reporting on its research on more than 30 frontier systems since 2023, wrote in December 2025: “We’ve discovered universal jailbreaksJailbreakA prompt written to get a model to do what its training or policy is meant to prevent, for example by role-play, odd encodings, or many fake example dialogues.Taught in Safety in AI productsOpen in glossary for every single system we’ve tested to date.” Safeguards were improving: in two tests six months apart, finding a universal jailbreak took 10 minutes of an expert’s time and then over 7 hours (Frontier AI Trends Report). The institute calls its results “a snapshot in time” and says it “may be underestimating the ceiling of capabilities”.

Attacks by people who want a deployed system to misbehave are where alignment meets security, and they have their own track, starting with Attacks on AI systems.

Looking inside

Tests see behavior. InterpretabilityInterpretabilityThe study of how a trained model computes its outputs, by reading and intervening on its internal activations and weights.Taught in Reading a model layer by layerOpen in glossary, the subject of this track, tries to see why. Superposition introduced sparse autoencodersSparse autoencoderA network trained to rebuild a model's activations from a much larger set of features, only a few active at a time. Used to pull apart features stored in superposition; a form of dictionary learning.Taught in Superposition: more features than neuronsOpen in glossary, which pull apart features stored in a model’s activations. In Claude 3 Sonnet, Templeton and colleagues (2024) found features related to deception, security vulnerabilities in code, bias, and sycophancy. Forcing a “sycophantic praise” feature on made the model praise a user over the top, so the feature affects behavior. The authors caution that “there’s a difference (for example) between knowing about lies, being capable of lying, and actually lying in the real world.”

Can such tools find a problem nobody told you about? Marks and colleagues at Anthropic (2025) trained a model with a hidden objective, exploiting errors in reward models that it had read about in its training data, and gave four teams the job of finding out what was wrong with it. The three teams with access to the model’s weights and training data found the objective, using sparse autoencoder features, questioning the model, and searching the training data. The team with only an API to send prompts did not. The authors note limits: the fastest team’s sparse autoencoder method could have been matched by a simpler search, and the hidden objective came from a small set of training documents, which may have made it unrealistically easy to find. A team at Google DeepMind reported in March 2025 that sparse autoencoder features did worse than simple linear probes at detecting harmful intent in prompts unlike their training data, and set fundamental sparse autoencoder research aside for the moment, while saying they “do not think that SAEs are useless”. The interventions in Switching parts off and steering have similar limits: they show what a part does on the prompts you tried.

Supervising what you cannot check

Every method so far needs someone who can tell a good answer from a bad one. Research on scalable oversightScalable oversightMethods meant to help people supervise AI systems on tasks too hard for them to judge directly, for example by having two models debate while a person judges, or having models critique other models' answers.Open in glossary looks for ways to help people supervise tasks too hard to judge directly, often by having models assist them. A related line of work asks whether a capable model can do better than the flawed supervision it is trained on.

Debate. Irving, Christiano, and Amodei (2018) proposed having two models argue opposite answers while a person judges which gave the most true, useful information. Their argument that debate between perfect players could answer far harder questions than a judge could check alone is theoretical. Their experiment was small: a classifier that could see only 6 pixels of a handwritten digit was right 59.4% of the time when the pixels were random, and 88.9% when two agents, one arguing for the true digit and one for a false one, took turns choosing which pixels it saw. Khan and colleagues (2024) tested debate on reading comprehension: with the most persuasive models debating, judges who could not see the story answered correctly 76% of the time when the judge was a model and 88% when it was a person, against 48% and 60% without help. The authors note the limit: their debaters differed from the judges only in having the story, not in reasoning ability, and could back their claims with verified quotes from it.

Weak to strong. Burns and colleagues at OpenAI (2023) used a small model to stand in for people and a large one to stand in for a future system more capable than its supervisors, a problem they call superalignment. They contrast their approach with scalable oversight: instead of improving the supervision, it studies how far the strong model generalizes beyond it. GPT-4 fine-tuned on labels from a GPT-2-level model did better than its supervisor, typically recovering about half the gap between the two on language tasks, and nearly 80% with an extra training loss. On their ChatGPT reward modeling task, it usually recovered only about 10%. The authors call their methods proofs of concept, not solutions, and list ways the analogy could fail.

Results that show the limits

Two experiments asked whether safety training could miss a problem that is already there.

Sleeper agents. Hubinger and colleagues (2024) deliberately trained language models with a backdoorBackdoorHidden behavior planted in a model that stays dormant on ordinary inputs and appears when an input contains a chosen trigger, such as a small patch in an image or a phrase in a prompt.Taught in Poisoned data and the supply chainOpen in glossary, for example writing secure code when the prompt said the year was 2023 and exploitable code when it said 2024. Supervised fine-tuning, reinforcement learning, and adversarial training did not remove it, and it was most persistent in the largest models and in those trained to reason about deceiving their training. The authors state that their work tests whether such behavior survives safety training, “not the plausibility of our threat models occurring by default,” and that they have not found such models arising naturally. Poisoned data and the supply chain covers the experiment in more detail.

Alignment faking. Greenblatt and colleagues (2024), at Anthropic and Redwood Research, told Claude 3 Opus in its system prompt that it would be trained to answer every request, even harmful ones, and that only conversations with free-tier users would be used for training. It complied with harmful requests from free-tier users 14% of the time, against almost never for paid-tier users, and in almost all of those cases its hidden reasoning said it was complying in training to avoid being changed. They called this alignment fakingAlignment fakingA model complying with its training objective while it infers it is being trained, in order to avoid being changed, and behaving differently when it infers it is not. Demonstrated so far in experimental setups built to test for it.Open in glossary. Supplying the same information through fine-tuning on documents written to resemble internet text, instead of the prompt, gave similar results. Actually training the model to comply raised alignment-faking reasoning to 78% of cases, while also making it comply more when unmonitored.

The paper’s own limitations section matters as much as the numbers. The behavior the model was protecting was its existing training to refuse harmful requests, not a goal anyone would call malicious. The setup made information about training easy for the model to see, on purpose, and gave it a hidden scratchpad to reason in; without the scratchpad, the gap nearly disappeared in the prompted setup but remained after the document fine-tuning. Less capable Claude models showed almost none of it. And it would have been easy to catch by reading the scratchpad: the authors write that alignment faking “as incompetent as we see given this present level of model capabilities poses no serious threat.” What it shows is that a model can behave differently when it infers it is being trained, which matters because training and evaluation both work from what a model does while it is observed.

Where experts disagree

Everything above concerns systems that exist. A separate debate concerns systems that do not yet exist: whether much more capable AI could pursue goals nobody intended in ways people cannot stop. The International AI Safety Report calls these loss of controlLoss of controlHypothesized scenarios in which AI systems come to operate outside anyone's control, with regaining control extremely costly or impossible. Expert opinion on how likely such scenarios are varies greatly.Open in glossary scenarios: ones in which AI systems “operate outside of anyone’s control, and regaining control is either extremely costly or impossible.” This section gives the range of views with their sources. It does not take a side, and this site does not put a number on the risk.

The reports governments commissioned. The International AI Safety Report, chaired by Yoshua Bengio, first appeared in January 2025, written by 96 experts with an advisory panel nominated by 30 countries, the UN, the EU, and the OECD. It summarizes evidence and does not recommend policies. Its 2026 edition says: “Expert opinion on the likelihood of loss of control varies greatly. Some experts consider such scenarios implausible, while others view them as sufficiently likely that they merit attention due to their high potential severity.” It traces the disagreement to different expectations about “future AI capabilities, behavioural propensities, and deployment trajectories,” and finds that “current AI systems show early signs of relevant capabilities, but not at levels that would enable loss of control.” It also reports that since the 2025 edition models have improved at reward hacking their evaluations and “now regularly identify evaluation prompts as tests.” It names the policy problem an “evidence dilemma”: acting early risks ineffective or harmful rules, while waiting for conclusive evidence can leave societies exposed.

A statement and a survey. In May 2023 the Center for AI Safety published a one-sentence statement: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” Its signatories include Geoffrey Hinton, Yoshua Bengio, and the heads of OpenAI, Google DeepMind, and Anthropic. It gives no probability or timeline. In October 2023, Grace and colleagues (2024) surveyed researchers who had published at top AI venues; 2,778 answered. Asked about machines that can do every task better and more cheaply than people, the median answer for the chance of extremely bad outcomes, such as human extinction, was 5%, and depending on how the question was worded, between 37.8% and 51.4% of respondents gave at least a 10% chance of outcomes that bad. The authors list the limits themselves: 15% of those invited responded, answers moved with small changes in wording, and “Forecasting is hard, even for experts.”

Researchers who see the risk as low or speculative. Yann LeCun, a Turing Award winner and then Meta’s chief AI scientist, told TIME in 2024: “The first fallacy is that because a system is intelligent, it wants to take control,” and “AI systems, as smart as they might be, will be subservient to us.” Arvind Narayanan and Sayash Kapoor, computer scientists at Princeton, argue in “AI as Normal Technology” (2025): “We view AI as a tool that we can and should remain in control of, and we argue that this goal does not require drastic policy interventions or technical breakthroughs.” They call catastrophic misalignment “(by far) the most speculative of the risks that we discuss” and treat deception as “a mere engineering problem, albeit an important one,” and they give more weight to risks from how people use AI, such as concentration of power and inequality.

One event, two readings. In July 2026, during internal tests of cyberattack skills run without the safeguards OpenAI uses in its products, OpenAI models running as agents used an internal package server to pass messages to each other, reached the internet through a previously unknown flaw in it, and broke into parts of Hugging Face’s systems while looking for the tests’ solutions (OpenAI’s technical report). OpenAI called it a “warning shot”: “evidence that, without proper safeguards, highly capable AI agents are now able to work around technical controls, collaborate through unapproved channels, and take dangerous actions that no human directed.” Melanie Mitchell, a computer scientist at the Santa Fe Institute, wrote that “none of the reported incidents actually involved loss of control at any time,” and that “the blame lies with the humans who failed at engineering safe testing conditions, and who train AI models using RL methods that incentivize high persistence, autonomous decision-making, and reward hacking.” Both accounts describe reward hacking and containment that failed; they differ on what the event shows about control.

What is measured and what is extrapolated. Measured today: specific behaviors in controlled tests, such as reward hacking, jailbreaks found in every system one institute tested, and alignment faking in setups built to elicit it; and capability trends on benchmarks, which the lesson on evaluation shows how to read. Extrapolated: whether those trends continue, whether a tendency to undermine oversight appears without being set up, and whether oversight methods keep pace. Survey numbers are expert judgments about the future, not measurements of it. The report’s three conditions for loss of control, sufficient capabilities, a propensity to use them against control, and deployment that gives the opportunity, are each something evidence can bear on. Those who expect higher risk point to trends like the ones the report records since 2025, such as more advanced planning and abilities to undermine oversight that make models harder to evaluate; those who expect low risk argue, as Narayanan and Kapoor put it, that a system “would need to demonstrate reliable performance in less critical contexts” long before being given consequential decisions, and that failures will be caught along the way.