Watch a network reshape space

A network with two neurons per layer, so every step is a 2D picture. Gridlines show what happens to the whole plane.

Challenges

0 of 3 complete

Turn the activation off (None) and play XOR or Circle to the last stage. Can stacked linear layers separate the classes? (not yet)
With ReLU, stop on an activation stage where at least half the points are pressed flat onto an axis. (not yet)
Untangle Circle: reach the last stage with a network that gets at least 98% of the training points right. (not yet)

Linear stages can only rotate, stretch, shear, and shift the grid: straight lines stay straight. Activation stages bend it.

Try this

Play XOR with Tanh, then with None. Without an activation the classes never come apart.
Try ReLU and look for regions folded flat against an axis.
Drag the stage slider slowly through the activation stages to see the gridlines bend.

Linear layers collapse, ReLU foldsPython with NumPy

import numpy as np

# XOR: opposite corners share a class, so no straight line separates them.
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y = np.array([0, 1, 1, 0])

# Two linear layers with nothing between them...
W1, b1 = np.array([[1.0, -1.0], [-1.0, 1.0]]), np.array([0.0, 0.0])
W2, b2 = np.array([[2.0, -1.0], [0.5, 3.0]]), np.array([0.1, -0.2])
two = (X @ W1.T + b1) @ W2.T + b2
# ...are exactly one linear layer, with weights W2 W1 and bias W2 b1 + b2.
one = X @ (W2 @ W1).T + (W2 @ b1 + b2)
print("two linear layers equal one:", np.allclose(two, one))

# Put ReLU after the first layer: negative coordinates become 0,
# folding those points flat onto an axis.
Z = X @ W1.T + b1
H = np.maximum(0, Z)
print("before ReLU:", Z.tolist())
print("after ReLU: ", H.tolist())

# In the new space one straight line separates the classes: h1 + h2 > 0.5.
pred = (H.sum(axis=1) > 0.5).astype(int)
print("predicted:", pred.tolist(), "true:", y.tolist())

two linear layers equal one: True
before ReLU: [[0.0, 0.0], [-1.0, 1.0], [1.0, -1.0], [0.0, 0.0]]
after ReLU:  [[0.0, 0.0], [0.0, 1.0], [1.0, 0.0], [0.0, 0.0]]
predicted: [0, 1, 1, 0] true: [0, 1, 1, 0]