Which model is better?

Two models with known true accuracy take the same benchmark. Their scores move from run to run, and sometimes the weaker one wins.

Stronger model (true 73.0%)
Weaker model (true 70.0%)

Curves: how likely each score is. Dotted lines: the true accuracies. Markers: the last run, with its 95% interval.

Questions in the benchmark100

True gap between the models3 points

Stronger model scorednot run yet

Weaker model scorednot run yet

Chance the weaker model ties or wins34.7%

95% interval on a 70% score±9.0 points

Your runs: weaker model tied or won0 of 0

Challenges

0 of 3 complete

With a gap of 3 points or more, run the benchmark until the weaker model scores at least as high as the stronger one. (not yet)
For a 3-point gap, find the smallest benchmark on the slider where the weaker model ties or wins less than 5% of the time. (not yet)
Make the 95% interval on a 70% score narrower than plus or minus 1 point. (not yet)

Try this

Run the 100-question benchmark ten times and count the upsets.
Find the smallest benchmark where a 3-point gap is reliable.
Set the gap to 1 point and try every size.

Error bars and upsets, computed exactlyPython with NumPy

import numpy as np
from math import lgamma, log

def half_width(score, n):
    """95% interval half-width: 1.96 standard errors of an accuracy."""
    return 1.96 * np.sqrt(score * (1 - score) / n)

def binomial(n, p):
    """Probability of each score 0..n when each answer is right with probability p."""
    k = np.arange(n + 1)
    logc = np.array([lgamma(n + 1) - lgamma(i + 1) - lgamma(n - i + 1) for i in k])
    return np.exp(logc + k * log(p) + (n - k) * log(1 - p))

def upset(n, weak=0.70, strong=0.73):
    """Exact chance that the weaker model scores at least as high as the stronger."""
    pw, ps = binomial(n, weak), binomial(n, strong)
    weak_at_least = np.cumsum(pw[::-1])[::-1]  # P(weak score >= s)
    return np.sum(ps * weak_at_least)

for n in [100, 164, 1000, 2000]:
    print(f"{n:5d} questions: 95% interval at 50% is +/-{100 * half_width(0.5, n):.1f} points, "
          f"weaker model ties or wins {upset(n):.1%} of the time")

  100 questions: 95% interval at 50% is +/-9.8 points, weaker model ties or wins 34.7% of the time
  164 questions: 95% interval at 50% is +/-7.7 points, weaker model ties or wins 29.4% of the time
 1000 questions: 95% interval at 50% is +/-3.1 points, weaker model ties or wins 7.2% of the time
 2000 questions: 95% interval at 50% is +/-2.2 points, weaker model ties or wins 1.9% of the time