Logistic regression
Shading shows the model's probability for each spot: blue for class A, orange for class B, gray where it is unsure. Tap to add points.
- Class A
- Class B
- Decision boundary (50%)
- 10% and 90% lines
Steps0
Mean cross-entropy0.6931
Accuracy50%
Weights w1, w2, b0, 0, 0
Challenges
0 of 3 complete
- On Separated data, keep training after every point is right until the mean cross-entropy is below 0.005. (not yet)
- Switch to Overlapping and train until the loss stops going down. How low does it get? (not yet)
- On Separated data, add at least three class B points deep inside the class A cloud, then train until the loss stops going down. (not yet)
Try this
- Train on Separated data and watch the 10% and 90% lines close in on the boundary.
- Switch to Overlapping and see where the loss levels off.
- Add points of one class inside the other’s region and retrain.
Logistic regression in NumPyPython with NumPy
import numpy as np
# Each point has two inputs (x1, x2) and a label: 0 = class A, 1 = class B.
X = np.array([[-1.5, -0.5], [-1.0, -1.2], [-0.6, 0.1], [-1.8, 0.4],
[1.2, 0.8], [0.9, 1.5], [0.4, 0.3], [1.7, -0.2]])
y = np.array([0, 0, 0, 0, 1, 1, 1, 1])
def sigmoid(z):
return 1 / (1 + np.exp(-z))
w = np.zeros(2) # weights w1, w2
b = 0.0 # bias
lr = 0.5 # step size
# The points are separable: watch the loss and weights after accuracy hits 100%.
for step in range(1, 1001):
p = sigmoid(X @ w + b) # each point's probability of class B
# Gradient of the mean cross-entropy: mean of (p - y) times each input.
w -= lr * X.T @ (p - y) / len(y)
b -= lr * np.mean(p - y)
if step in (1, 10, 100, 1000):
p = sigmoid(X @ w + b)
loss = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))
right = np.mean((p >= 0.5) == y)
print(f"step {step:4d}: loss {loss:.4f}, accuracy {right:.0%}, "
f"w1 {w[0]:.2f}, w2 {w[1]:.2f}, b {b:.2f}")step 1: loss 0.5255, accuracy 100%, w1 0.28, w2 0.11, b 0.00
step 10: loss 0.1710, accuracy 100%, w1 1.43, w2 0.57, b 0.01
step 100: loss 0.0407, accuracy 100%, w1 3.42, w2 1.26, b 0.07
step 1000: loss 0.0070, accuracy 100%, w1 6.72, w2 2.07, b 0.23The one-input version, with sliders for the weight and bias:
From score to probability
Made-up data: hours of practice before a driving test, and whether the test was passed. The curve is the model's probability of passing.
- Failed (y = 0)
- Passed (y = 1)
- Gap to the right answer (thickest: the costliest point)
Mean cross-entropy0.378
Accuracy at 50%81%
50% point2.5 hours