Neural network playground
Train a real network in your browser. Shading shows its prediction everywhere; small squares show what each neuron computes.
Blue regions are predicted blue, orange regions orange. Paler means less sure.
Tap a neuron to show its output full size. Lines are weights: solid red positive, dashed blue negative, thicker is larger.
Epoch0
Training loss0.000
Test loss0.000
Test accuracy0%
Parameters0
Challenges
0 of 4 complete
- Raise the learning rate until training breaks: get the training loss above 1, worse than a 50/50 guess (0.69). (not yet)
- Separate the circle with at most 3 hidden neurons in total: reach 95% test accuracy. (not yet)
- Untangle the spiral: reach 90% test accuracy. (not yet)
- Make it overfit: keep training until the test loss has climbed 0.1 above its lowest point while the training loss keeps falling. (not yet)
Try this
- Train the default circle, then set hidden layers to 0. A network with no hidden layer can only draw a straight line.
- Try the spiral with ReLU, 2 layers of 8, and a learning rate near 0.01.
- Set noise to 1 with 8 neurons per layer and train for a few hundred epochs. Watch test loss rise while training loss falls.
- Tap any neuron in the diagram to see what it computes.
The training loop in NumPyPython with NumPy
import numpy as np
# Twelve points: six inside a circle (class 0), six on a ring around it (class 1).
angles = np.linspace(0, 2 * np.pi, 6, endpoint=False)
inner = 0.3 * np.c_[np.cos(angles), np.sin(angles)]
outer = 0.9 * np.c_[np.cos(angles + 0.5), np.sin(angles + 0.5)]
X = np.vstack([inner, outer])
y = np.array([0] * 6 + [1] * 6)
# 2 inputs -> 4 tanh neurons -> 2 outputs, starting from small random weights.
rng = np.random.default_rng(0)
W1, b1 = rng.normal(0, np.sqrt(1 / 2), (2, 4)), np.zeros(4)
W2, b2 = rng.normal(0, np.sqrt(1 / 4), (4, 2)), np.zeros(2)
def forward(X):
h = np.tanh(X @ W1 + b1)
logits = h @ W2 + b2
p = np.exp(logits - logits.max(axis=1, keepdims=True))
return h, p / p.sum(axis=1, keepdims=True) # softmax probabilities
lr, batch = 0.5, 4 # plain SGD (the demo also offers Adam), minibatches of 4
for epoch in range(301):
if epoch % 100 == 0:
_, p = forward(X)
loss = -np.log(p[np.arange(len(y)), y]).mean() # cross-entropy
accuracy = (p.argmax(axis=1) == y).mean()
print(f"epoch {epoch}: loss {loss:.3f}, accuracy {accuracy:.0%}")
order = rng.permutation(len(y)) # shuffle, then one step per minibatch
for start in range(0, len(y), batch):
i = order[start:start + batch]
h, p = forward(X[i])
d_logits = p.copy()
d_logits[np.arange(len(i)), y[i]] -= 1 # gradient of the loss at the logits
d_logits /= len(i)
d_h = (d_logits @ W2.T) * (1 - h**2) # backpropagate through tanh
W2 -= lr * h.T @ d_logits
b2 -= lr * d_logits.sum(axis=0)
W1 -= lr * X[i].T @ d_h
b1 -= lr * d_h.sum(axis=0)epoch 0: loss 0.734, accuracy 50%
epoch 100: loss 0.088, accuracy 100%
epoch 200: loss 0.019, accuracy 100%
epoch 300: loss 0.010, accuracy 100%