Learn a gridworld by trial and error

The agent (orange dot) knows nothing about the world. Each cell's four triangles show its estimate of each move's return; the dashed line is the route it currently thinks is best.

Episodes0

Exploration now0.300

Average return, last 20 episodesn/a

Learned routegoes in circles (some moves still untried)

Challenges

0 of 3 complete

In the maze, train until the learned route (the dashed line) reaches the +1 by the shortest path: 9 moves. (not yet)
On the cliff, get Q-learning to learn the shortest route: 13 moves, right along the edge. (not yet)
On the cliff, with exploration that does not fade (epsilon at least 0.05), get SARSA to learn a route that keeps at least one row away from the edge. (not yet)

Try this

Press Train at Watch speed. The first episodes wander; the first time the agent stumbles onto the +1, only the cell next to it learns anything.
Switch to Fast and watch the dashed route settle on the shortest path.
Switch to the cliff, train Q-learning to 500 episodes, then press Forget, switch to SARSA, and train again. Compare the routes and the return chart.
Set exploration to 0 on a fresh maze. Does the agent still explore?

Q-learning versus SARSA on a small cliffPython with NumPy

import numpy as np

# A small cliff: start S bottom left, goal G bottom right, cliff C between.
# Every move costs 1; stepping onto the cliff costs 100 more and ends the episode.
H, W = 3, 6
start, goal = (2, 0), (2, 5)
cliff = {(2, c) for c in range(1, 5)}
moves = [(-1, 0), (0, 1), (1, 0), (0, -1)]  # up, right, down, left

def step(s, a):
    r, c = s[0] + moves[a][0], s[1] + moves[a][1]
    nxt = (min(max(r, 0), H - 1), min(max(c, 0), W - 1))
    if nxt in cliff:
        return nxt, -101, True
    return nxt, -1, nxt == goal

def train(method, episodes=500, alpha=0.5, epsilon=0.1, seed=0):
    rng = np.random.default_rng(seed)
    Q = np.zeros((H, W, 4))
    def choose(s):  # epsilon-greedy, ties broken at random
        if rng.random() < epsilon:
            return int(rng.integers(4))
        q = Q[s]
        return int(rng.choice(np.flatnonzero(q == q.max())))
    for _ in range(episodes):
        s, a, done = start, choose(start), False
        while not done:
            s2, r, done = step(s, a)
            a2 = choose(s2)
            if done:
                target = r
            elif method == "q-learning":
                target = r + Q[s2].max()   # value of the best next action
            else:
                target = r + Q[s2][a2]     # value of the action it will take
            Q[s][a] += alpha * (target - Q[s][a])
            s, a = s2, a2
    return Q

def greedy_route(Q):
    s, route = start, [start]
    while s != goal and len(route) < 30:
        s, _, _ = step(s, int(np.argmax(Q[s])))
        route.append(s)
    return route

for method in ["q-learning", "sarsa"]:
    route = greedy_route(train(method))
    rows = sorted({r for r, _ in route})
    print(f"{method:10s}: {len(route) - 1} moves, uses rows {rows}")

q-learning: 7 moves, uses rows [1, 2]
sarsa     : 9 moves, uses rows [0, 1, 2]