Five slot machines

Each machine pays out with its own hidden chance. Three strategies play the same machines with the same luck.

Marker: share of pulls that paid. Bar: where the true chance probably lies (90% belief).

Pulls0 of 1000

Epsilon-greedy regret0.0

UCB regret0.0

Thompson sampling regret0.0

Thompson sampling pulled the best machinen/a

Challenges

0 of 3 complete

Set epsilon to 0 (never explore) and run at least 300 pulls. Can pure greed lock onto a machine that is not the best? (not yet)
Turn on the average of 200 runs, then tune UCB's bonus weight c until UCB ends with less average regret than Thompson sampling. (not yet)
Find luck (press New luck) where Thompson sampling, one of the strongest strategies here, ends 1,000 pulls with regret above 60. (not yet)

Try this

Press Play and watch the regret chart. Which line keeps climbing at a steady slope?
Set epsilon to 0 and press New luck a few times. How often does pure greed settle on the wrong machine?
Show what UCB believes. Untried and rarely tried machines get long bars: that is the optimism UCB acts on.
Turn on the average of 200 runs and slide UCB’s c between 0 and 2.

Greedy, UCB, and Thompson sampling on two machinesPython with NumPy

import numpy as np

# After 14 pulls: machine A paid 2 of its 4 pulls, machine B 7 of its 10.
pulls = np.array([4, 10])
wins = np.array([2, 7])
t = pulls.sum()

# Epsilon-greedy and UCB both track the share of pulls that paid.
means = wins / pulls

# UCB adds a bonus that is large for machines tried only a few times.
c = np.sqrt(2)  # UCB1
bonus = c * np.sqrt(np.log(t) / pulls)
ucb = means + bonus

# Thompson sampling keeps a belief per machine: Beta(1 + wins, 1 + losses).
a, b = 1 + wins, 1 + pulls - wins
belief_mean = a / (a + b)
belief_sd = np.sqrt(a * b / ((a + b) ** 2 * (a + b + 1)))

for name, m, bo, u, bm, bs in zip("AB", means, bonus, ucb, belief_mean, belief_sd):
    print(f"{name}: mean {m:.2f} + bonus {bo:.2f} = UCB score {u:.2f}"
          f" | belief {bm:.2f} +/- {bs:.2f}")
print("Greedy pulls", "AB"[np.argmax(means)], "next; UCB pulls", "AB"[np.argmax(ucb)])

# Thompson sampling draws one plausible chance per machine and pulls the higher.
rng = np.random.default_rng(0)
draws = rng.beta(a, b, size=(100_000, 2))
print(f"Thompson picks A in {np.mean(draws[:, 0] > draws[:, 1]):.0%} of draws")

A: mean 0.50 + bonus 1.15 = UCB score 1.65 | belief 0.50 +/- 0.19
B: mean 0.70 + bonus 0.73 = UCB score 1.43 | belief 0.67 +/- 0.13
Greedy pulls B next; UCB pulls A
Thompson picks A in 24% of draws