Random forest

Many deep trees, each trained on a bootstrap sample of the points, vote on every spot. Shading shows the share of trees voting class B.

Class A
Class B

Gray shading means the trees are split; strong color means they agree. Dashed rings mark labels flipped on purpose.

Challenges

0 of 3 complete

Make the forest beat a single fully grown tree by at least 5 points of test accuracy. (not yet)
Switch to one tree and find a member whose test accuracy is at least 10 points below the forest's. (not yet)
Find the fewest trees for which every training point has been left out of at least one tree's sample. (not yet)

Try this

Slide the number of trees from 1 to 100 and watch the boundary smooth out.
Switch to One tree and step through a few members. How different are they?
Turn the random feature off to get plain bagging, and compare the test accuracy.
Watch the out-of-bag line on the chart: it is computed from training points only.

Bootstrap samples and why votes helpPython with NumPy

import numpy as np
from math import comb

rng = np.random.default_rng(0)

# A bootstrap sample: n draws with replacement from n points.
n = 150
left_out = [n - len(np.unique(rng.integers(0, n, n))) for _ in range(2000)]
print(f"share left out of a bootstrap sample: {np.mean(left_out) / n:.3f}")
print(f"(1 - 1/n)^n = {(1 - 1 / n) ** n:.3f}, 1/e = {np.exp(-1):.3f}")

# Why voting helps: if each of k voters is right 70% of the time and they
# err independently, how often is the majority right?
for k in (1, 5, 25, 51):
    majority = sum(comb(k, r) * 0.7**r * 0.3**(k - r) for r in range(k // 2 + 1, k + 1))
    print(f"{k:3d} independent voters: majority right {majority:.3f}")

share left out of a bootstrap sample: 0.366
(1 - 1/n)^n = 0.367, 1/e = 0.368
  1 independent voters: majority right 0.700
  5 independent voters: majority right 0.837
 25 independent voters: majority right 0.983
 51 independent voters: majority right 0.999