Getting less wrong

Imagine you run an ice cream stand. On hot days you sell more, on cool days fewer, and you would like to predict tomorrow’s sales from the weather forecast so you buy the right amount of cream. You have notes from sixteen past days: the temperature and how many cones you sold.

The simplest prediction is a straight line: the hotter it is, the more you sell. But which line? There are infinitely many. To pick one, you first need a way to say how good a line is.

Measuring a miss

For any line, look at each past day. The line predicts some number of cones for that day’s temperature, and you know how many you really sold. The difference is the miss for that day.

One miss per day is a lot to keep track of. To compare lines, we want a single number. Adding up the misses does not work: a line that predicts 20 too many on one day and 20 too few on another would add up to zero, as if it were perfect.

The standard fix is to square each miss before averaging. Squaring makes every miss positive, so they cannot cancel. It also makes big misses count extra: a miss of 20 counts four times as much as a miss of 10. The result is called the mean squared errorMean squared errorThe average of the squared differences between predictions and true values. Squaring makes big misses count much more than small ones.Taught in Linear regression and lossOpen in glossary. Take its square root and you are back in cones, which gives you a “typical miss” you can read directly.

A number that measures how wrong a model is, like this one, is called a lossLoss functionA formula that turns a model's mistakes on the training data into a single number. Lower is better, and training tries to make it as low as possible.Taught in The learning loopOpen in glossary. Lower is better.

import numpy as np

# The sixteen days from the demo: temperature (°C) and cones sold.
temp = np.array([26, 27, 19, 23, 29, 24, 14, 15, 19, 30, 17, 30, 15, 26, 18, 31])
cones = np.array([176, 140, 118, 160, 196, 136, 78, 92, 123, 214, 81, 171, 36, 146, 97, 209])

# Measure temperature from the average day (about 23°C). The best line is
# the same, but tilt and height stop interfering, so small steps work well.
t = temp - temp.mean()

def typical_miss(slope, height):
    """Square each day's miss, average the squares, take the square root."""
    predicted = slope * t + height
    return np.sqrt(np.mean((predicted - cones) ** 2))

# Start with a flat line: the same sales whatever the weather.
slope, height = 0.0, 136.0
print(f"flat line: misses by {typical_miss(slope, height):.1f} cones")

# Gradient descent: work out which way each number should move to lower
# the error, move it a small step that way, and repeat.
for step in range(300):
    miss = slope * t + height - cones
    slope -= 0.02 * np.mean(2 * miss * t)
    height -= 0.02 * np.mean(2 * miss)

print(f"after 300 steps: misses by {typical_miss(slope, height):.1f} cones")
print(f"predicted sales at 30°C: {slope * (30 - temp.mean()) + height:.0f} cones")

flat line: misses by 49.3 cones
after 300 steps: misses by 18.4 cones
predicted sales at 30°C: 194 cones

Learning is getting less wrong, one step at a time

When you dragged the handles, you were doing what the computer does, only by feel. The computer has no eyes, so it does something methodical. At each step it asks: if I tilted the line slightly, or moved it up or down slightly, would the error go up or down? Then it moves the line a small amount in whichever direction lowers the error, and asks again.

This is called gradient descentGradient descentAn optimization method that repeatedly moves the parameters a small step in the direction that lowers the loss fastest, the opposite of the gradient.Taught in Gradient descentOpen in glossary, and it is how almost every modern AI model learns, including the ones behind chatbots. The model there has billions of adjustable numbers instead of two, but the loop is the same:

1. Make predictions with the current settings.
2. Measure the loss.
3. Nudge every setting a little in the direction that lowers the loss.
4. Repeat, often thousands or even millions of times.

The gradient descent lesson in the machine learning track looks at this loop in detail, including what can go wrong when the steps are too big or too small.

The best line still misses

Even the best line misses by about 18 cones on a typical day. That is not a failure. Sales depend on more than temperature: weekends, rain, a school trip passing by. A straight line through temperature alone cannot know about any of that.

You could draw a wiggly curve that passes through every single point, with zero error on these sixteen days. It would be useless for tomorrow, because it would be following the accidents of those particular days rather than the real pattern. A good model captures the trend and accepts some error on the examples it learned from.