The standard network is the 784-64-32-10 network from the reading handwriting lesson. The adversarially trained one has the same shape and was trained on attacked digits: every training batch was replaced by a 7-step attack at epsilon 0.1. Every attack on the single digit runs in your browser; the step attacks’ curves over 200 digits were computed ahead with PyTorch, and tests check this page’s attack reproduces them.
Fool the handwriting network
Move every pixel a little in the direction that most raises the network's loss, and compare with random noise of the same size.
Loading two networks and 500 test digits (about 820 KB)...
True digit...
Answer before...
Answer after...
Largest pixel change...
Challenges
0 of 4 complete
- Take a digit the network reads correctly with over 99% confidence and change its answer, moving no pixel by more than 0.1. (not yet)
- With the standard network, find an epsilon where random noise leaves over 95% of the curve digits correct but one gradient step leaves under 60%. (not yet)
- Switch to the adversarially trained network and find an epsilon where it beats the standard network by over 60 points under one gradient step. (not yet)
- On the adversarially trained network, find an epsilon where ten small steps beat one big step by over 10 points. (not yet)
Try this
- Compare one gradient step with random noise at the same epsilon.
- Find a digit read at over 99% and flip it with epsilon 0.1 or less.
- Switch networks and compare the curves, including at epsilon 0.
- On the adversarially trained network, compare one step with ten small steps at epsilon 0.2.
Gradient sign against random signs, on a logistic regressionPython with NumPy
import numpy as np
rng = np.random.default_rng(1)
n, d = 400, 100 # 400 examples with 100 "pixels" each
# Two classes whose average pictures differ a little in every pixel.
y = rng.integers(0, 2, n)
X = rng.normal(0, 1, (n, d)) + np.where(y[:, None] == 1, 0.3, -0.3)
# Logistic regression, trained by gradient descent.
w, b = np.zeros(d), 0.0
for _ in range(500):
p = 1 / (1 + np.exp(-(X @ w + b)))
w -= 0.1 * X.T @ (p - y) / n
b -= 0.1 * np.mean(p - y)
def accuracy(inputs):
return np.mean((inputs @ w + b > 0) == y)
def input_gradient(inputs):
# Cross-entropy loss: its gradient with respect to the input is (p - y) w.
p = 1 / (1 + np.exp(-(inputs @ w + b)))
return (p - y)[:, None] * w[None, :]
eps = 0.3
fgsm = X + eps * np.sign(input_gradient(X))
noise = X + eps * rng.choice([-1, 1], size=X.shape)
print(f"clean accuracy: {accuracy(X):.1%}")
print(f"random signs, eps 0.3: {accuracy(noise):.1%}")
print(f"FGSM, eps 0.3: {accuracy(fgsm):.1%}")
# Why: FGSM moves the score w.x by eps times the sum of |w|, in the worst direction;
# random signs mostly cancel, moving it by about eps times the length of w.
print(f"FGSM shifts the score by {eps * np.abs(w).sum():.2f}, noise by about {eps * np.linalg.norm(w):.2f}")clean accuracy: 100.0%
random signs, eps 0.3: 100.0%
FGSM, eps 0.3: 49.8%
FGSM shifts the score by 6.74, noise by about 0.70