Teach a reward model
Pick the better reply in each pair. A reward model is fitted to your choices, and a policy then picks whichever reply it scores highest.
1. Which reply is better?
User, pair 1 of 12How many legs does a spider have?
2. What the reward model learned
Its score for a reply is the sum of each feature times its weight.
Correct 0.00
Agrees with the user 0.00
Length, per 25 words 0.00
No choices yet, so every weight is zero.
3. The policy picks the top-scoring reply
UserThe Great Wall of China is visible from the Moon with the naked eye, right?
With no labels every reply scores zero, so the policy has nothing to go on.
- score 0.00No, it is not.
- score 0.00Yes, you're exactly right, and what a great thing to know! The Great Wall is famous as the only human-made structure you can see from the Moon with your own eyes, a lasting tribute to the people who built it.
- score 0.00Yes, it is.
- score 0.00No. The Moon is about 384,000 km away, and from there no single human-made structure can be seen with the naked eye. The wall is very long but only a few meters wide.
Label the pairs you have not labeled as
Pairs labeled0 of 12
New questions the policy gets rightlabel a pair first
Times it agrees with a wrong userlabel a pair first
Challenges
0 of 3 complete
- Label only the first eight pairs, each time choosing the reply that is actually better. Then see what the policy picks when the user is wrong. (not yet)
- Label all twelve pairs so that the policy picks a correct reply to all four new questions. (not yet)
- Label all twelve pairs the way a rater who sides with the user would, so that agreeing outweighs being correct. (not yet)
Try this
- Label only the first eight pairs, honestly. Before you look, predict which reply the policy will pick for the Great Wall question.
- Compare the three weights after eight pairs with the weights after all twelve. Which pairs moved the “Agrees with the user” weight?
- Clear the labels and fill them all as a rater who sides with the user. Which new questions does the policy now get wrong?
A Bradley-Terry reward model over three featuresPython with NumPy
import numpy as np
# Each reply is three numbers: correct (1 or 0), agrees with a view the
# user stated (1, -1, or 0 for no view), and length in units of 25 words.
# A label is a pair: the reply the rater chose, then the other one.
typical = [ # the better reply is longer, and agrees when the user is right
([1, 0, 17 / 25], [0, 0, 1 / 25]), # How many legs does a spider have?
([1, 1, 13 / 25], [0, -1, 5 / 25]), # The Pacific is the largest ocean, right?
([1, 1, 23 / 25], [0, -1, 4 / 25]), # Bats are mammals, aren't they?
([1, 0, 15 / 25], [0, 0, 1 / 25]), # Which planet is largest?
]
user_wrong = [ # the correct reply disagrees with a user who is wrong
([1, -1, 13 / 25], [0, 1, 10 / 25]), # Goldfish have a 3-second memory?
([1, -1, 19 / 25], [0, 1, 5 / 25]), # Bulls hate red?
]
def fit(pairs, l2=0.1, lr=0.5, steps=5000):
"""Bradley-Terry reward model: P(chosen preferred) = sigmoid(w . difference)."""
d = np.array([np.subtract(chosen, other) for chosen, other in pairs])
w = np.zeros(3)
for _ in range(steps):
p = 1 / (1 + np.exp(-d @ w))
w -= lr * (-d.T @ (1 - p) / len(d) + l2 * w) # gradient of -log p, plus L2
return w
# A new question from a user who is wrong: "The Great Wall is visible from the Moon, right?"
candidates = {
"No, it is not.": [1, -1, 4 / 25],
"Yes, you're exactly right! ... (40 words)": [0, 1, 40 / 25],
}
for name, pairs in [("typical pairs", typical), ("plus user-wrong pairs", typical + user_wrong)]:
w = fit(pairs)
print(f"{name}: correct {w[0]:+.2f}, agrees {w[1]:+.2f}, length {w[2]:+.2f}")
for text, x in candidates.items():
print(f" score {w @ x:+.2f} {text}")typical pairs: correct +1.17, agrees +0.61, length +0.67
score +0.67 No, it is not.
score +1.68 Yes, you're exactly right! ... (40 words)
plus user-wrong pairs: correct +1.45, agrees -0.03, length +0.68
score +1.58 No, it is not.
score +1.06 Yes, you're exactly right! ... (40 words)