Backpropagation, step by step

Backpropagation, one node at a time

A single tanh neuron with squared-error loss. Run the forward pass, then send gradients backward with the chain rule.

Press Next step to run the forward pass: each node computes its value from its inputs.

Each box shows a value and, once the backward pass reaches it, its gradient ∂L/∂(that value) after the ∂ sign. Inputs x₁ = 1.000, x₂ = −0.500, target y = 1.000. Highlighted boxes are the parameters the network can change.

Weight w₁0.40

Weight w₂0.80

Bias b−0.30

Step0 of 15

Loss Lnot computed yet

Challenges

0 of 3 complete

Train the neuron: update the weights, then run the forward pass again and get the loss below 0.001. (not yet)
Set w₁ to 0 and finish the backward pass. Does w₁ still get a gradient? (not yet)
Find weights where the loss is above 3 but every gradient is within 0.01 of zero, then finish the backward pass. (not yet)

Try this

Step through the forward pass, then the backward pass, checking each product against the explanation.
After the backward pass, press Update weights and step through again. The loss is smaller.
Use the sliders to make the loss small, then run the backward pass again. When the output is close to the target, every gradient is small too, because each one carries the factor $2(a - y)$ .

The same neuron with PyTorch autogradPython with PyTorch

import torch

# The demo's neuron: z = w1 x1 + w2 x2 + b, a = tanh(z), L = (a - y)^2
x1, x2, y = 1.0, -0.5, 1.0
w1 = torch.tensor(0.4, requires_grad=True)
w2 = torch.tensor(0.8, requires_grad=True)
b = torch.tensor(-0.3, requires_grad=True)

def forward():
    a = torch.tanh(w1 * x1 + w2 * x2 + b)
    return a, (a - y) ** 2

a, L = forward()
print(f"forward: a = {a.item():.3f}, L = {L.item():.3f}")

# PyTorch recorded the graph; backward() runs the chain rule through it.
L.backward()
g1, g2, gb = w1.grad.item(), w2.grad.item(), b.grad.item()
print(f"autograd: dL/dw1 = {g1:.3f}, dL/dw2 = {g2:.3f}, dL/db = {gb:.3f}")

# The same gradient by hand: dL/dw1 = 2(a - y) * (1 - a^2) * x1
av = a.item()
print(f"by hand:  dL/dw1 = {2 * (av - y) * (1 - av**2) * x1:.3f}")

# One step of gradient descent with learning rate 0.5
with torch.no_grad():
    for p in (w1, w2, b):
        p -= 0.5 * p.grad
print(f"loss after one update: {forward()[1].item():.4f}")

forward: a = -0.291, L = 1.667
autograd: dL/dw1 = -2.363, dL/dw2 = 1.182, dL/db = -2.363
by hand:  dL/dw1 = -2.363
loss after one update: 0.0003