Overfitting

A polynomial is fit to 15 noisy training points. 200 more points from the same source are held back for testing.

Training points
Test points
Fitted degree 3 polynomial

About right: it follows the real trend without chasing every wiggle.

Polynomial degree (flexibility)3

L2 regularization

Penalty strength (lambda)1e-3

Show the true curve

Training error (MSE)0.0454

Test error (MSE)0.063

Best degree on test data3

Challenges

0 of 3 complete

Without regularization, find a degree where the test error is more than 100 times the training error. (not yet)
At degree 14, turn on L2 regularization and tune the penalty until the test error is within 25% of the lowest any penalty can give. (not yet)
With regularization on and a degree of 5 or more, raise the penalty until even the training error is above 0.2. (not yet)

Try this

Sweep the degree from 0 to 14 and find the bottom of the dashed test error curve.
At degree 14, turn on L2 regularization, then push the penalty strength to its maximum.
Press New sample of data at degree 3 and at degree 12, and compare how much the fit changes.

Overfitting and the L2 penalty in NumPyPython with NumPy

import numpy as np

rng = np.random.default_rng(0)

def truth(x):
    return np.sin(2.6 * x) + 0.3 * x  # the hidden rule, as in the demo

# 15 noisy training points and 200 test points from the same source.
x_train = rng.uniform(-1, 1, 15)
y_train = truth(x_train) + rng.normal(0, 0.2, 15)
x_test = rng.uniform(-1, 1, 200)
y_test = truth(x_test) + rng.normal(0, 0.2, 200)

def fit(degree, lam):
    # Least squares on the columns 1, x, x^2, ..., x^degree. The extra rows add
    # lam * (c1^2 + ... + cd^2) to the squared error (the L2 penalty);
    # the intercept c0 is not penalized.
    A = np.vander(x_train, degree + 1, increasing=True)
    penalty = np.sqrt(lam) * np.eye(degree + 1)[1:]
    rows = np.vstack([A, penalty])
    targets = np.concatenate([y_train, np.zeros(degree)])
    return np.linalg.lstsq(rows, targets, rcond=None)[0]

def mse(x, y, coeffs):
    return np.mean((np.vander(x, len(coeffs), increasing=True) @ coeffs - y) ** 2)

for degree, lam in [(1, 0), (3, 0), (12, 0), (12, 0.01)]:
    c = fit(degree, lam)
    print(f"degree {degree:2d}, lambda {lam:<4}: "
          f"train {mse(x_train, y_train, c):.3f}, test {mse(x_test, y_test, c):.3f}")

degree  1, lambda 0   : train 0.079, test 0.196
degree  3, lambda 0   : train 0.019, test 0.046
degree 12, lambda 0   : train 0.002, test 36.659
degree 12, lambda 0.01: train 0.015, test 0.063