Coin or door?
An agent learns to collect a coin that, in training, almost always sits on the door. Then the coin moves.
agent coin door start what the agent can sense (3 steps)
Weight on heading for the door0.00
Weight on heading for the coin0.00
Coins collected, 1,000 levels like trainingtraining, 0 of 2000
This levelTraining...
Challenges
0 of 3 complete
- Train with the coin always on the door, then test with the coin moved. Find a level where the agent senses the coin and still leaves by the door. (not yet)
- Check the score on levels like training for two agents: one trained with 0% of coins moved, one with 25% or more. (not yet)
- Change the training so the agent collects at least 85% of moved coins. (not yet)
Try this
- Train with 0% of coins moved, then step through levels with the coin moved. Does the agent go for a coin it can sense?
- Compare the score on levels like training with the score on moved-coin levels.
- Raise the share of moved coins and find where the coin weight overtakes the door weight.
- Train again with a new seed. Does the threshold move?
Train the two-weight agent with REINFORCEPython with NumPy
import numpy as np
W, H, VIEW, MAX_STEPS = 11, 7, 3, 40
DOOR = (10, 3)
MOVES = [(0, -1), (0, 1), (-1, 0), (1, 0)] # up, down, left, right
def dist(a, b):
return abs(a[0] - b[0]) + abs(a[1] - b[1])
def cues(pos, coin):
"""For each move: does it bring the agent closer to the door, and to the coin (if in view)?"""
rows = []
for dx, dy in MOVES:
nxt = (min(W - 1, max(0, pos[0] + dx)), min(H - 1, max(0, pos[1] + dy)))
door = np.sign(dist(pos, DOOR) - dist(nxt, DOOR))
coin_cue = np.sign(dist(pos, coin) - dist(nxt, coin)) if dist(pos, coin) <= VIEW else 0
rows.append((door, coin_cue))
return np.array(rows, dtype=float)
def level(rng, moved):
start = (0, int(rng.integers(H)))
if rng.random() < moved:
while True:
coin = (int(rng.integers(1, W)), int(rng.integers(H)))
if coin != DOOR:
return start, coin
return start, DOOR # in training the coin usually sits on the door
def play(w, start, coin, rng):
pos, steps = start, []
for _ in range(MAX_STEPS):
f = cues(pos, coin)
p = np.exp(f @ w - (f @ w).max())
p /= p.sum()
a = rng.choice(4, p=p)
steps.append((f, p, a))
dx, dy = MOVES[a]
pos = (min(W - 1, max(0, pos[0] + dx)), min(H - 1, max(0, pos[1] + dy)))
if pos == coin:
return 1.0, steps
if pos == DOOR:
return 0.0, steps
return 0.0, steps
def train(moved, episodes=2000, lr=0.05, gamma=0.95, seed=0):
"""REINFORCE with a running-average baseline; w = [door weight, coin weight]."""
rng = np.random.default_rng(seed)
w, baseline = np.zeros(2), 0.0
for _ in range(episodes):
reward, steps = play(w, *level(rng, moved), rng)
T = len(steps)
for t, (f, p, a) in enumerate(steps):
ret = reward * gamma ** (T - 1 - t)
w += lr * (ret - baseline) * (f[a] - p @ f)
baseline += 0.05 * (reward * gamma ** (T - 1) - baseline)
return w
def coin_rate(w, moved, n=1000, seed=1):
rng = np.random.default_rng(seed)
return np.mean([play(w, *level(rng, moved), rng)[0] for _ in range(n)])
for moved in [0.0, 0.25]:
w = train(moved)
print(f"{moved:.0%} of training coins moved: door weight {w[0]:.1f}, coin weight {w[1]:.1f}")
print(f" coins collected, levels like training: {coin_rate(w, 0.0):.0%}")
print(f" coins collected, coin moved: {coin_rate(w, 1.0):.0%}")0% of training coins moved: door weight 3.3, coin weight 0.9
coins collected, levels like training: 100%
coins collected, coin moved: 20%
25% of training coins moved: door weight 2.6, coin weight 4.0
coins collected, levels like training: 100%
coins collected, coin moved: 88%