Train a policy directly

REINFORCE adjusts one number, the chance of choosing right, after every episode. The orange dot shows where the policy sits on the curve of expected return.

Every step costs 1, and all three cells look the same to the agent.

Start from

Mostly left (0.1)Mostly right (0.9)

Learning rate0.0050

Subtract a baseline (running average of returns)

Fast

Episodes0

Chance of right, p0.100

Expected return−42.22

Last episoden/a

Challenges

0 of 3 complete

Train until the policy's expected return beats −12. (The best any nearly certain policy can do is −44.) (not yet)
Raise the learning rate until a single update moves the chance of going right by more than 0.5. (not yet)
Start from mostly right (0.9) and train until the expected return beats −12. (not yet)

Try this

Press Train and watch the orange dot climb the curve. Where does it stop?
Turn the learning rate all the way up and train again. Watch the chart of p for sudden jumps.
Start from mostly right and train. Does it end up in the same place?
Turn the baseline off at a medium learning rate and compare how jittery the chart of p is.

REINFORCE with a baseline on the short corridorPython with NumPy

import numpy as np

# The short corridor: cells 0, 1, 2, then the goal. Every step costs 1.
# In cell 1 the actions are switched. The agent cannot tell the cells apart,
# so its whole policy is one number: p, the chance of choosing "right".
def run_episode(p, rng, max_steps=1000):
    cell, actions = 0, []
    while cell < 3 and len(actions) < max_steps:
        right = rng.random() < p
        actions.append(right)
        forward = (not right) if cell == 1 else right
        cell = cell + 1 if forward else max(0, cell - 1)
    return actions

def J(p):  # exact expected return from the start
    return 2 * (p - 2) / (p * (1 - p))

rng = np.random.default_rng(0)
theta, alpha, baseline = np.log(0.1 / 0.9), 0.005, 0.0  # start at p = 0.1
for episode in range(1, 301):
    p = 1 / (1 + np.exp(-theta))
    actions = run_episode(p, rng)
    T = len(actions)
    grad = 0.0
    for t, right in enumerate(actions):
        G = -(T - t)                       # return from step t on
        score = (1 - p) if right else -p   # d/dtheta of ln pi(action)
        grad += (G - baseline) * score
    for t in range(T):                     # running average of returns
        baseline += 0.05 * (-(T - t) - baseline)
    theta += alpha * grad
    if episode % 50 == 0:
        p = 1 / (1 + np.exp(-theta))
        print(f"episode {episode}: p(right) = {p:.2f}, expected return {J(p):.1f}")

print(f"best possible: p = {2 - np.sqrt(2):.3f}, expected return {J(2 - np.sqrt(2)):.2f}")

episode 50: p(right) = 0.56, expected return -11.7
episode 100: p(right) = 0.51, expected return -11.9
episode 150: p(right) = 0.58, expected return -11.7
episode 200: p(right) = 0.60, expected return -11.7
episode 250: p(right) = 0.62, expected return -11.7
episode 300: p(right) = 0.52, expected return -11.9
best possible: p = 0.586, expected return -11.66