A pairwise arena

Voters compare two anonymous models at a time. Turn the votes into ratings, and watch the ratings and their uncertainty settle.

Rating with 95% interval
True rating

Bradley-Terry: all votes fit at once by maximum likelihood. Intervals: 100 bootstrap resamples of the votes. Ties are left out.

Rating method

Bradley-TerryOnline Elo

Voters favor long answers (Model E writes the longest)

Votes0 of 6,000

Order matches the true orderno votes yet

Largest change from the last shufflenot shuffled

Challenges

0 of 3 complete

With ordinary voters and Bradley-Terry, collect votes until no two neighboring models' 95% intervals overlap. (not yet)
Switch to online Elo, collect at least 500 votes, then shuffle the vote order. Make a rating move by more than 10 points. (not yet)
Switch on voters who favor long answers and collect votes until Model E, truly the weakest, ranks in the top three. (not yet)

Try this

Add votes 100 at a time with Bradley-Terry and watch the intervals shrink.
Compare online Elo and Bradley-Terry after 100 votes, then after 3,000.
Shuffle the vote order under each method.
Turn on voters who favor long answers.

Online Elo versus Bradley-Terry on the same votesPython with NumPy

import numpy as np

# Each vote is (winner, loser) among three models: 0, 1, 2.
votes = [(0, 1), (0, 2), (1, 2), (0, 1), (2, 1), (1, 0), (0, 2), (1, 2), (0, 1), (2, 0)]

def expected(ra, rb):
    """Elo: chance that A beats B."""
    return 1 / (1 + 10 ** ((rb - ra) / 400))

def online_elo(votes, k=32):
    r = np.full(3, 1000.0)
    for w, l in votes:
        d = k * (1 - expected(r[w], r[l]))
        r[w] += d
        r[l] -= d
    return r

def bradley_terry(votes, iters=200):
    """Maximum likelihood with the classic iterative update (Zermelo; Hunter 2004)."""
    wins = np.zeros((3, 3))
    for w, l in votes:
        wins[w, l] += 1
    games = wins + wins.T
    s = np.ones(3)  # strengths; P(i beats j) = s_i / (s_i + s_j)
    for _ in range(iters):
        s = wins.sum(1) / (games / (s[:, None] + s[None, :])).sum(1)
        s /= np.exp(np.log(s).mean())
    return 1000 + 400 * np.log10(s)  # Elo scale, mean log strength at 1000

print("online Elo, original order:", np.round(online_elo(votes)).astype(int).tolist())
print("online Elo, reversed order:", np.round(online_elo(votes[::-1])).astype(int).tolist())
print("Bradley-Terry, either order:", np.round(bradley_terry(votes)).astype(int).tolist(),
      np.round(bradley_terry(votes[::-1])).astype(int).tolist())

online Elo, original order: [1034, 989, 977]
online Elo, reversed order: [1045, 986, 969]
Bradley-Terry, either order: [1109, 974, 917] [1109, 974, 917]