Training a network

You now have every piece. A network is layers of matrix multiplications with activations in between. A loss measures how wrong it is. Backpropagation computes the gradient of that loss for every weight, and gradient descent nudges every weight downhill. Repeat that thousands of times and the network learns. This lesson puts it all in your hands.

The training loop

Training a network on a dataset looks like this:

1. Start every weight at a small random value.
2. Shuffle the training examples and split them into small groups called minibatches.
3. For each minibatch: run the forward pass, compute the loss, backpropagate, and update every weight.
4. When every minibatch has been used once, one epochEpochOne full pass through the training data. Training usually runs for many epochs, shuffling the examples each time.Open in glossary is done. Go back to step 2.

Using a minibatch instead of the whole dataset for each step is stochastic gradient descent: each step’s gradient is a noisy estimate of the true one, but the steps are cheap and frequent. The batch sizeBatch sizeHow many training examples are used to compute each gradient step. Small batches give noisy but frequent updates; large batches give smoother but fewer updates per epoch.Open in glossary sets that trade.

The network below trains exactly this way, in your browser. Each point is an example with two inputs, its coordinates. The big square shows the network’s prediction for every possible input. The diagram shows every neuron as a small square of its own: what that neuron outputs for every point on the plane.

import numpy as np

# Twelve points: six inside a circle (class 0), six on a ring around it (class 1).
angles = np.linspace(0, 2 * np.pi, 6, endpoint=False)
inner = 0.3 * np.c_[np.cos(angles), np.sin(angles)]
outer = 0.9 * np.c_[np.cos(angles + 0.5), np.sin(angles + 0.5)]
X = np.vstack([inner, outer])
y = np.array([0] * 6 + [1] * 6)

# 2 inputs -> 4 tanh neurons -> 2 outputs, starting from small random weights.
rng = np.random.default_rng(0)
W1, b1 = rng.normal(0, np.sqrt(1 / 2), (2, 4)), np.zeros(4)
W2, b2 = rng.normal(0, np.sqrt(1 / 4), (4, 2)), np.zeros(2)

def forward(X):
    h = np.tanh(X @ W1 + b1)
    logits = h @ W2 + b2
    p = np.exp(logits - logits.max(axis=1, keepdims=True))
    return h, p / p.sum(axis=1, keepdims=True)  # softmax probabilities

lr, batch = 0.5, 4  # plain SGD (the demo also offers Adam), minibatches of 4
for epoch in range(301):
    if epoch % 100 == 0:
        _, p = forward(X)
        loss = -np.log(p[np.arange(len(y)), y]).mean()  # cross-entropy
        accuracy = (p.argmax(axis=1) == y).mean()
        print(f"epoch {epoch}: loss {loss:.3f}, accuracy {accuracy:.0%}")
    order = rng.permutation(len(y))  # shuffle, then one step per minibatch
    for start in range(0, len(y), batch):
        i = order[start:start + batch]
        h, p = forward(X[i])
        d_logits = p.copy()
        d_logits[np.arange(len(i)), y[i]] -= 1  # gradient of the loss at the logits
        d_logits /= len(i)
        d_h = (d_logits @ W2.T) * (1 - h**2)  # backpropagate through tanh
        W2 -= lr * h.T @ d_logits
        b2 -= lr * d_logits.sum(axis=0)
        W1 -= lr * X[i].T @ d_h
        b1 -= lr * d_h.sum(axis=0)

epoch 0: loss 0.734, accuracy 50%
epoch 100: loss 0.088, accuracy 100%
epoch 200: loss 0.019, accuracy 100%
epoch 300: loss 0.010, accuracy 100%

What each setting does

Depth and width set the network’s capacity: how complicated a boundary it can form. Zero hidden layers gives a straight line. One hidden layer of a few neurons can enclose a region. Spirals need more neurons and usually more layers, because each layer can build on the shapes the previous one made.

The activation function shapes the pieces. ReLU networks build boundaries out of straight segments, so their predictions have corners. Tanh and sigmoid networks build smooth curves. With no activation, any number of layers is still a single straight line, as you saw in the previous lesson on nonlinearity.

The learning rateLearning rateThe step size in gradient descent. Too small and training crawls; too large and each step overshoots, so the loss can grow instead of shrink.Taught in Gradient descentOpen in glossary is the size of each step. Too large and steps overshoot, so the loss bounces or blows up. Too small and training crawls. Adam adapts the step size for each weight separately based on the history of its gradients, which is why it copes with a wider range of learning rates than plain SGD. The optimizers lesson shows why.

Random starting weights matter more than you might expect. Press New starting weights a few times on the spiral: some starts find a good solution quickly, others take longer or get stuck. Starting at random is also essential. If every neuron in a layer began identical, every one would get the same gradient and they would stay identical forever.

Watching for overfitting

The dashed line in the loss chart is the test loss, computed on points the network never trains on. Turn on Show test data to see them as hollow marks. As long as the two curves fall together, the network is learning patterns that hold for new data.

That is overfittingOverfittingWhen a model fits the quirks and noise of its training data so closely that it does worse on new data. Training error keeps falling while test error rises.Taught in Overfitting and generalizationOpen in glossary: with enough capacity, the network starts fitting the noise in its particular training points. The usual remedies are more data, a smaller network, regularization, or simply stopping when test loss stops improving. The overfitting lesson in the machine learning track covers them in depth.