A tiny neural network

Two inputs, a layer of hidden neurons, and an output. Train it and watch the shading: one neuron can only draw a straight line, but several together can bend.

Challenges

0 of 3 complete

Train 1 hidden neuron on Circle until it stops at 2,500 steps. How high does it get? (not yet)
Get 95% or more of the Circle points right with as few hidden neurons as possible. (not yet)
Solve XOR: get 95% or more right. Try 2 hidden neurons first. (not yet)

Try this

Train with 1 hidden neuron on Circle, then with 4. How many lines does it take to surround the middle?
Try XOR with 2 hidden neurons, then with 3 or 4.
Turn off the lines and watch only the shading while it trains.

Optional: the same idea in codePython with NumPy

import numpy as np

# Blue points (label 0) in a small circle in the middle,
# orange points (label 1) in a ring around them.
angle = np.linspace(0, 2 * np.pi, 12, endpoint=False)
circle = np.column_stack([np.cos(angle), np.sin(angle)])
X = np.vstack([0.3 * circle, 1.0 * circle])  # each row: across, up
y = np.array([0] * 12 + [1] * 12)

def train(hidden, steps=3000, learning_rate=1.0):
    rng = np.random.default_rng(1)
    # Each hidden neuron has 2 weights (one per input) and a bias.
    W1, b1 = rng.normal(size=(2, hidden)), np.zeros(hidden)
    # The output neuron has one weight per hidden neuron, and a bias.
    w2, b2 = rng.normal(size=hidden), 0.0
    for _ in range(steps):
        h = np.tanh(X @ W1 + b1)              # each hidden neuron's answer
        p = 1 / (1 + np.exp(-(h @ w2 + b2)))  # chance of orange, 0 to 1
        # Backpropagation: start from the error at the output (d_out), pass
        # it back through the output weights to get each hidden neuron's
        # share (d_hidden), then nudge every weight against its blame.
        # The demo uses a smarter nudging rule called Adam; the idea is the same.
        d_out = (p - y) / len(y)
        d_hidden = np.outer(d_out, w2) * (1 - h**2)
        w2 -= learning_rate * (h.T @ d_out)
        b2 -= learning_rate * d_out.sum()
        W1 -= learning_rate * (X.T @ d_hidden)
        b1 -= learning_rate * d_hidden.sum(axis=0)
    return np.mean((p > 0.5) == y)

for hidden in (1, 2, 3):
    print(f"Hidden neurons: {hidden}, points right: {train(hidden):.0%}")

Hidden neurons: 1, points right: 71%
Hidden neurons: 2, points right: 92%
Hidden neurons: 3, points right: 100%