Gradient descent

Gradient descent on a loss surface

Contour lines join points of equal loss; the stronger the tint, the lower the loss. Tap anywhere to start there.

One minimum, but steeper in one direction. Too large a step makes the path zig-zag.

Tap to choose a start

Tap the map to choose a starting point, or press Step. The arrow points downhill: the direction of the negative gradient.

Steps0

Loss11.22

Slope (gradient size)11.4978

Challenges

0 of 3 complete

On the Stretched bowl, get a run to converge using a step size of 0.2 or more the whole way. (not yet)
On Many valleys, tap different starting points until runs have settled in two different valleys. (not yet)
On Banana valley, run until the loss is below 0.001. Watch the shape of the loss chart. (not yet)

Try this

On the Stretched bowl, compare step sizes 0.1, 0.2, and 0.26.
On Many valleys, tap several starting points and see which valley each one finds.
On Banana valley, turn on Fast forward and watch the loss chart flatten into a long slide.

Gradient descent on the stretched bowl in NumPyPython with NumPy

import numpy as np

def loss(p):
    # The stretched bowl: it curves 8 times more steeply along y than along x.
    return 0.5 * (p[0] ** 2 + 8 * p[1] ** 2)

def gradient(p):
    # The slope in each direction (the partial derivatives of the loss).
    return np.array([p[0], 8 * p[1]])

def descend(lr, start=(-2.6, 1.4), max_steps=2000):
    p = np.array(start)
    for step in range(max_steps):
        g = gradient(p)
        if np.linalg.norm(g) < 1e-4:   # slope is essentially zero: done
            return f"converged after {step} steps, loss {loss(p):.1e}"
        if np.abs(p).max() >= 1000:    # flew off the map
            return f"diverged after {step} steps"
        p = p - lr * g                 # one step downhill
    return f"still going after {max_steps} steps"

# The limit is 2 / 8 = 0.25: just below it the path zig-zags but settles.
for lr in (0.01, 0.1, 0.2, 0.249, 0.26):
    print(f"step size {lr}: {descend(lr)}")

step size 0.01: converged after 1012 steps, loss 4.9e-09
step size 0.1: converged after 97 steps, loss 4.5e-09
step size 0.2: converged after 46 steps, loss 4.1e-09
step size 0.249: converged after 1448 steps, loss 6.2e-10
step size 0.26: diverged after 86 steps

The same idea with a single parameter:

Downhill in one dimension

The ball shows the current parameter value. Drag it to choose a starting point, then step.

The slope is negative (uphill to the left), so the step goes right. The arrow along the bottom shows the next step: step size times slope, 0.3 times -2.5.

Step size (learning rate)0.30

Loss curve

Simple bowlTwo dips

Steps0

Parameter x-2.5

Loss3.125

Slope-2.5