Plan with values

Each cell shows what it is worth: the reward you can expect from there on. Arrows point to the best move, and the dashed line is the planned route.

World

Classic 4 by 3Cliff edge

Reward for every move−0.04

Discount (gamma)1.00

Chance a move slips sideways20%

Sweeps28 (settled)

Largest change in the last sweepbelow 0.000001

Value of the start0.705

Planned route+1 exit in 5 moves

Challenges

0 of 4 complete

In the classic world, make the cell beside the −1 exit walk straight into the wall instead of moving up. (not yet)
In the classic world, make moving so costly that the cell beside the −1 exit jumps in on purpose. (not yet)
In the cliff edge world, make the route settle for the nearby +1 by the path along the cliff. (not yet)
In the cliff edge world, make the route go for the +10 along the cliff edge. (not yet)

Try this

Press Watch it sweep and watch value spread outward from the exits, one cell per sweep.
In the classic world, drag the reward for every move from −0.04 toward 0, then toward −2. Watch the cell beside the −1 exit change its mind.
In the cliff edge world, set the slip to 0%, then lower the discount.
Set the discount to 1 and the reward for every move above 0. Why do the values never settle?

Value iteration on the 4 by 3 worldPython with NumPy

import numpy as np

# The 4 by 3 world: "." empty, "#" wall, numbers are exits. Start bottom left.
rows = [". . . +1",
        ". # . -1",
        ". . . ."]
grid = [r.split() for r in rows]
H, W = len(grid), len(grid[0])
moves = [(-1, 0), (0, 1), (1, 0), (0, -1)]  # up, right, down, left
gamma, living, slip = 1.0, -0.04, 0.2

def land(r, c, a):
    """Where moving in direction a from (r, c) lands; walls and edges block."""
    nr, nc = r + moves[a][0], c + moves[a][1]
    if 0 <= nr < H and 0 <= nc < W and grid[nr][nc] != "#":
        return nr, nc
    return r, c

def q_value(V, r, c, a):
    """Expected reward plus discounted value: 80% as intended, 10% to each side."""
    total = 0.0
    for d, p in [(a, 1 - slip), ((a + 1) % 4, slip / 2), ((a + 3) % 4, slip / 2)]:
        nr, nc = land(r, c, d)
        cell = grid[nr][nc]
        if cell in (".", "#"):
            total += p * (living + gamma * V[nr, nc])
        else:  # an exit: its reward, and the episode ends
            total += p * (living + float(cell))
    return total

V = np.zeros((H, W))
for sweep in range(1, 1000):
    new = V.copy()
    for r in range(H):
        for c in range(W):
            if grid[r][c] == ".":
                new[r, c] = max(q_value(V, r, c, a) for a in range(4))
    change = np.abs(new - V).max()
    V = new
    if change < 1e-6:
        break

print(f"settled after {sweep} sweeps")
print(np.round(V, 3))
arrows = "^>v<"
for r in range(H):
    print(" ".join(arrows[int(np.argmax([q_value(V, r, c, a) for a in range(4)]))]
                   if grid[r][c] == "." else grid[r][c] for c in range(W)))

settled after 28 sweeps
[[0.812 0.868 0.918 0.   ]
 [0.762 0.    0.66  0.   ]
 [0.705 0.655 0.611 0.388]]
> > > +1
^ # ^ -1
^ < < <