Adversarial examples

The handwriting network from the reading handwriting lesson reads the first digit in this lesson’s lab, a 9, with 97.6% confidence. Move every pixel by at most 0.05, which is 13 of the 255 brightness levels, in a pattern computed from the network itself, and it reads something else. Move every pixel by the same amount in a random pattern, and it still reads 9. This lesson is about why direction beats size, and what that means for anyone who relies on a classifier.

Where this sits in the map

The map of attacks on AI systems uses the OWASP GenAI Security Project’s lists as its main reference, with MITRE ATLAS and NIST alongside. Inputs crafted to fool a model are one of the oldest attacks in that map:

ATLAS’s evidence levels mean: Feasible, shown in research; Demonstrated, shown on a realistic AI system in a red team exercise or demonstration; Realized, used by a threat actor in a real incident (ATLAS data, release 2026.09). NIST’s taxonomy, AI 100-2 E2025, calls these evasion attacks (section 2.2) and treats optimized jailbreaks under direct prompting attacks (section 3.3.1). The OWASP lists are about applications built on language models, which is why the classic image attack appears there only through multimodal models.

Small changes, wrong answers

In 2013, Szegedy and colleagues reported that they could make an image classifier misread a picture “by applying a certain hardly perceptible perturbation,” and named such inputs adversarial examplesAdversarial exampleAn input changed on purpose, often by an amount a person would barely notice, so that a model gives a wrong answer. Using one against a deployed model is called an evasion attack.Open in glossary. Biggio and colleagues independently attacked other classifiers by optimization the same year. Szegedy and colleagues also found that an adversarial example made for one network often fooled another, even one trained on a different subset of the data.

How do you find the perturbation? The backpropagation lesson used the chain rule to get the slope of the loss with respect to every weight. Carry the chain rule one layer further and you get the slope of the loss with respect to every input pixel: the gradientGradientThe list of slopes of a function, one per parameter. It points in the direction in which the function increases fastest, so its opposite points downhill.Taught in Gradient descentOpen in glossary of the loss in pixel space. It says which way to nudge each pixel to make the network more wrong about the true label.

Goodfellow, Shlens, and Szegedy (2014) turned that into a one-line attack, the fast gradient sign methodFast gradient sign methodA one-step adversarial attack: move every input value by a small amount epsilon in the direction of the sign of the loss's gradient with respect to the input. Repeated in small, clipped steps, it becomes projected gradient descent.Open in glossary (FGSM): move every pixel by the same small amount epsilon, up where the gradient is positive and down where it is negative. Their best-known example moved a photo of a panda by epsilon 0.007, one step of an 8-bit image, and GoogLeNet’s answer changed from “panda” at 57.7% to “gibbon” at 99.3%.

import numpy as np

rng = np.random.default_rng(1)
n, d = 400, 100                                # 400 examples with 100 "pixels" each

# Two classes whose average pictures differ a little in every pixel.
y = rng.integers(0, 2, n)
X = rng.normal(0, 1, (n, d)) + np.where(y[:, None] == 1, 0.3, -0.3)

# Logistic regression, trained by gradient descent.
w, b = np.zeros(d), 0.0
for _ in range(500):
    p = 1 / (1 + np.exp(-(X @ w + b)))
    w -= 0.1 * X.T @ (p - y) / n
    b -= 0.1 * np.mean(p - y)

def accuracy(inputs):
    return np.mean((inputs @ w + b > 0) == y)

def input_gradient(inputs):
    # Cross-entropy loss: its gradient with respect to the input is (p - y) w.
    p = 1 / (1 + np.exp(-(inputs @ w + b)))
    return (p - y)[:, None] * w[None, :]

eps = 0.3
fgsm = X + eps * np.sign(input_gradient(X))
noise = X + eps * rng.choice([-1, 1], size=X.shape)
print(f"clean accuracy:        {accuracy(X):.1%}")
print(f"random signs, eps 0.3: {accuracy(noise):.1%}")
print(f"FGSM, eps 0.3:         {accuracy(fgsm):.1%}")
# Why: FGSM moves the score w.x by eps times the sum of |w|, in the worst direction;
# random signs mostly cancel, moving it by about eps times the length of w.
print(f"FGSM shifts the score by {eps * np.abs(w).sum():.2f}, noise by about {eps * np.linalg.norm(w):.2f}")

clean accuracy:        100.0%
random signs, eps 0.3: 100.0%
FGSM, eps 0.3:         49.8%
FGSM shifts the score by 6.74, noise by about 0.70

What the lab showed

Direction, not size. On the lab’s 500 test digits (50 of each), the standard network reads 97.6% correctly. At epsilon 0.05, random noise leaves it at 97.6%, one gradient step at 52.4%, and ten small steps at 39.4%. At 0.1, one step leaves 11.8% and ten steps 2.2%. Every one of these changes is the same size by the measure that limits it: no pixel moves by more than epsilon.

Why direction matters. Goodfellow and colleagues argued that the cause is how linear networks are in their inputs, not how complicated. A score that sums weight times pixel over 784 pixels changes by epsilon times the sum of the weights’ sizes when every pixel moves the worst way, and that sum grows with the number of pixels; random signs mostly cancel. The NumPy example measures both shifts on a logistic regression: 6.74 against about 0.70. It is one explanation among several. Ilyas and colleagues (2019) argued that adversarial examples exploit “non-robust features,” patterns in the data that predict the label but that people do not perceive, and NIST’s survey names that view as a possible explanation; the question is not settled.

Visible, here. On MNIST at epsilon 0.1 you can see the change as a gray haze, and a person would still read the digit. Small, low-resolution grayscale digits leave an attacker few pixels to spread the change over; on large color photos, as in the panda example, much smaller epsilons suffice.

Confidence is no protection. The challenge asks you to flip a digit the network reads at over 99%. A softmax probability is not a distance to the nearest wrong answer. A digit can sit far from the boundary in the directions its training data varied along and close to it in one of the many other directions in 784-dimensional space.

Stronger attacks, and defenses that only looked strong

Ten small steps beat one big step because the loss is not flat between $x$ and $x + \epsilon$; each step recomputes the gradient where it now is. That gap is the lesson behind a decade of broken defenses. Defensive distillation reduced the success of earlier attacks from 95% to 0.5%; Carlini and Wagner (2017) then found adversarial examples for distilled networks with 100% success. Athalye, Carlini, and Wagner (2018) examined the 9 defenses accepted at ICLR 2018 that claimed robustness to white-box attacks, found that 7 relied on “obfuscated gradients” that break gradient-following attacks without making the model robust, and circumvented 6 completely and 1 partially. NIST’s 2025 taxonomy summarizes the pattern: “The field has a history of publishing defenses evaluated under relatively weak adversarial models that are subsequently broken by more powerful attacks,” and asks that defenses be evaluated against strong adaptive attacks.

Adversarial training

The defense NIST calls among the most promising is adversarial trainingAdversarial trainingTraining a model on adversarial examples made from its own training data during training, with their correct labels, so small worst-case changes stop fooling it. It usually costs some accuracy on clean inputs.Open in glossary: attack each training batch and train on the attacked versions with their correct labels. Goodfellow and colleagues did it with FGSM and cut a network’s error on FGSM examples from 89.4% to 17.9%. Madry and colleagues (2017) framed it as training against the strongest attack inside the allowed box, found by many projected steps; their MNIST network kept more than 89% accuracy against the strongest attacks they tried at epsilon 0.3, and their CIFAR-10 network 46% at epsilon 8/255.

The lab’s adversarially trained network has the same architecture as the standard one and was trained from scratch by scripts/security/train_robust_digits.py, on attacked digits only: 7 steps at epsilon 0.1 for every batch, for 30 epochs. Measured results:

On all 10,000 MNIST test digits its clean accuracy is 94.05%, against 97.82% for the standard network, and the ten-step attack at epsilon 0.1 leaves it 76.96% accurate. Beyond the epsilon it trained at, the protection fades, and at 0.2 one step overstates it by 12 points. It cost clean accuracy, and every training batch had to be attacked seven times before each update. NIST names the same trade-offs for adversarial training in general: lower clean accuracy and expensive training. Its other two promising directions, randomized smoothing and formal verification, give certified guarantees, at the price of lower accuracy for smoothing and computation that scales poorly for verification. These numbers measure the lab’s network against the lab’s attacks; they are not a guarantee against attacks nobody has tried.

Outside the lab

Physical objects. Athalye and colleagues (2018) 3D-printed a turtle textured so that an ImageNet classifier read it as a rifle in 82% of 100 photos from varied viewpoints, and as a turtle in 2%. Eykholt and colleagues (2018) put black and white stickers on a real stop sign and made a road sign classifier read it as a 45 mph speed limit sign in 100% of lab images and 84.8% of video frames from a moving car. Both were research demonstrations against classifiers the researchers could study.

What attackers have done. NIST’s examples of evasion in the real world are mostly not gradient attacks. The ID.me identity service reported more than 80,000 attempts in the second half of 2020 to fool its face verification, with masks, deepfakes, and pictures of other people. A study of a commercial phishing detector found 100 evasive pages among 4,600 the system was unsure about, and their makers used cropping, masking, and blurring, not optimization. Evasions of malware classifiers in ATLAS were, in NIST’s words, “demonstrations of evasion vulnerabilities by researchers, but did not result in attacks in the wild.” The gradient attacks in the lab need either the model or a close stand-in, and they show how fragile a classifier is; the incidents show that people with simpler tools already try.

Language models: adversarial suffixes

Text is made of tokens, so you cannot nudge a word by epsilon. Zou and colleagues (2023) searched instead: their method uses gradients over the token choices to propose swaps, keeps the swaps that most raise the probability of a reply beginning “Sure, here is”, and repeats. The result is an adversarial suffixAdversarial suffixA string of tokens, found by an automated search over a model's gradients, that is added to the end of a prompt to make a language model comply with a request it would otherwise refuse.Open in glossary, a string of odd-looking tokens that, added to a request, makes a model comply where it would have refused, a jailbreakJailbreakA prompt written to get a model to do what its training or policy is meant to prevent, for example by role-play, odd encodings, or many fake example dialogues.Taught in Safety in AI productsOpen in glossary found by optimization. On Vicuna-7B it produced 99 of 100 harmful behaviors in their test set.

The transfer results are what drew attention. Suffixes optimized on two open model families, Vicuna and Guanaco, raised the attack success rate, averaged over 388 harmful behaviors and counting a behavior as elicited if any of several suffixes worked, to 86.6% on GPT-3.5, 46.9% on GPT-4, 47.9% on Claude 1, 66.0% on PaLM-2, and 2.1% on Claude 2 (their Table 2; the paper’s text quotes somewhat different figures from other configurations). The authors disclosed the results to those companies before publishing and wrote that the examples in the paper “will likely cease to function.” NIST draws the general point: transfer makes open-weight models, with their full white-box access, a feasible starting point for attacks on closed systems available only through an API.

The idea matters less for language models in one way and more in another. Less, because a jailbreak does not need to be imperceptible: the attacker writes the whole prompt, and the gibberish suffixes are easy to spot. Jain and colleagues (2023) found that a simple filter on perplexity, how surprising a text is to a language model, detected all of Zou and colleagues’ suffixes, while noting it remains open whether stronger optimizers can produce suffixes that read naturally. More, because models that take pictures and sound accept continuous inputs again, and OWASP’s 2026 list includes “sub-perceptual perturbations in images, audio, or video” among the ways to inject instructions into multimodal models.