Why a baseline helps
Each mark is one episode's estimate of the gradient at the same policy. Subtracting a baseline from the returns changes how the estimates spread, not where they center.
True gradient15.50
Expected return at this policy−22.5
No baselineaverage 20.2, spread 76.2
With baselineaverage 20.2, spread 76.2
Challenges
0 of 3 complete
- Find the baseline value that makes the estimates spread the least (within 1% of the smallest possible spread). (not yet)
- Choose a baseline that makes the estimates spread at least 1.5 times as wide as with no baseline. Where does their average go? (not yet)
- Find the policy where the true gradient is zero. (not yet)
Try this
- Press Set b to the expected return and compare the two rows. Which is narrower?
- Drag the baseline to +40. The marks spread out, but where is their average?
- Move the policy to 0.59. The true gradient is zero; what do single estimates look like?
The log-derivative trick, checked exactlyPython with NumPy
import numpy as np
# One choice between two actions: action 0 pays 1, action 1 pays 3.
# The policy picks action 1 with probability p = sigmoid(theta).
rewards = np.array([1.0, 3.0])
theta = 0.4
p = 1 / (1 + np.exp(-theta))
probs = np.array([1 - p, p])
# The expected reward and its true gradient: dJ/dtheta = (3 - 1) p (1 - p).
J = probs @ rewards
true_grad = (rewards[1] - rewards[0]) * p * (1 - p)
# REINFORCE estimates the gradient from one sampled action A as
# (reward - b) * d/dtheta ln pi(A). For this policy the score is
# 1 - p for action 1 and -p for action 0.
score = np.array([-p, 1 - p])
print(f"p = {p:.3f}, true gradient = {true_grad:.4f}")
print("average score over actions is zero:", bool(np.isclose(probs @ score, 0)))
for b in [0.0, 2.0, J, 6.0]:
estimates = (rewards - b) * score # one estimate per possible action
mean = probs @ estimates # exact average over actions
spread = np.sqrt(probs @ (estimates - mean) ** 2)
print(f"baseline {b:4.2f}: average {mean:.4f}, spread {spread:.4f}")p = 0.599, true gradient = 0.4805
average score over actions is zero: True
baseline 0.00: average 0.4805, spread 0.8836
baseline 2.00: average 0.4805, spread 0.0967
baseline 2.20: average 0.4805, spread 0.1935
baseline 6.00: average 0.4805, spread 2.0574