Gradient boosting

Start from the average, then repeatedly fit a small tree to what is still wrong (the residuals) and add a fraction of it.

Challenges

0 of 3 complete

With learning rate 1 and stumps, find the round where test error is lowest. (not yet)
Get the test error below 0.12 using stumps. (not yet)
Drive the training error below 0.001. (not yet)

Try this

Press Add one tree a few times and compare the step in the lower panel with the change in the model above.
Set the learning rate to 0.1 and watch how slowly the model bends toward the points.
Run 300 trees at learning rate 1 and compare the two error curves.
Switch to depth 2 trees and see how quickly the training error reaches zero.

Gradient boosting with stumpsPython with NumPy

import numpy as np

x = np.array([0.05, 0.15, 0.25, 0.35, 0.45, 0.55, 0.65, 0.75, 0.85, 0.95])
y = np.array([0.4, 0.9, 1.1, 0.7, 0.2, -0.5, -0.9, -0.6, 0.1, 0.6])

def fit_stump(x, r):
    # The threshold whose two side means fit the residuals r best.
    best = (np.inf, None)
    for t in (x[:-1] + x[1:]) / 2:
        left, right = r[x <= t], r[x > t]
        sse = ((left - left.mean())**2).sum() + ((right - right.mean())**2).sum()
        if sse < best[0]:
            best = (sse, (t, left.mean(), right.mean()))
    return best[1]

learning_rate = 0.5
F = np.full(len(x), y.mean())  # start from the average
for m in range(1, 31):
    residuals = y - F               # the negative gradient of squared error / 2
    t, lo, hi = fit_stump(x, residuals)
    F += learning_rate * np.where(x <= t, lo, hi)
    if m in (1, 2, 5, 10, 30):
        print(f"after {m:2d} stumps: training MSE {np.mean((y - F)**2):.4f}")

after  1 stumps: training MSE 0.2447
after  2 stumps: training MSE 0.1907
after  5 stumps: training MSE 0.0937
after 10 stumps: training MSE 0.0444
after 30 stumps: training MSE 0.0068