Learning by trial and error
Some problems have no answer key, only feedback. Balance exploring against exploiting, watch an agent learn to cross a gridworld from rewards alone, and see how the same ideas fine-tune language models and why they can be gamed.
- Level
- Intermediate
- Length
- 5 lessons, 2 hr
- Assumes
- The machine learning and neural networks tracks.
- Builds on
- How machines learn and Neural networks
- 1Explore or exploit: banditsWith no answer key, a learner has to try things to find out what pays. Race three classic strategies on five slot machines and measure what exploring costs.20 min
- 2States, actions, and rewardsWhen actions change your situation, rewards can arrive long after the choices that earned them. Values and the Bellman equation turn that into a plan.25 min
- 3Learning from experience: Q-learningNo map, no model, just moves and rewards. Watch an agent learn a maze from scratch with temporal difference updates, and see why Q-learning and SARSA walk a cliff differently.25 min
- 4Policy gradientsSkip the values and adjust the policy itself, nudging up the chance of whatever worked. Train REINFORCE live and see why a baseline makes it steadier without making it wrong.25 min
- 5Rewarding LLMs, and reward hackingHow reinforcement learning fine-tunes language models from human comparisons and automatic checks, and why optimizing any measured reward too hard can make the real goal worse.25 min
Labs in this track
- Five slot machinesRace epsilon-greedy, UCB, and Thompson sampling on the same slot machines with the same luck, and watch regret pile up.
- Plan with valuesRun value iteration on two gridworlds and watch how the cost of moving, the discount, and slippery moves change the best route.
- Learn a gridworld by trial and errorWatch Q-learning and SARSA learn a maze and a cliff walk from rewards alone, one move at a time, with no map of the world.
- Train a policy directlyTrain REINFORCE live on a corridor where the best policy is random, and watch learning rate and baseline shape the climb.
- Why a baseline helpsSee a thousand REINFORCE gradient estimates at once, and slide a baseline to watch their spread change while their average stays put.
- Overoptimizing a reward modelPush a model toward a flawed reward model's favorite answers with best-of-n or a KL penalty, and watch true quality rise, peak, and fall.