Voting over sampled answers

Majority vote over sampled answers

If each sampled chain of thought is right with probability p, how often is the most common answer among n samples right?

One sample is right (p)40%

Samples per question (n)9

Possible answers

2 answers4, spread4, one tempting

One sample right40%

Vote of 9 right (exact)59.9%

Simulated, 1,000 votesNot run yet

Challenges

0 of 3 complete

Find a setting where one sample is right less than half the time, yet the vote is right at least 90% of the time. (not yet)
Make voting backfire: one sample is right at least 30% of the time, but the vote is right less than 10% of the time. (not yet)
Find a setting where a few samples help but 41 samples do worse than one. (not yet)

Voting helps: the correct answer is the most likely single answer. Setup: four options, wrong answers spread evenly.

Real reasoning traces: DeepSeek-R1-Distill-Qwen-1.5B

Sixteen sampled traces per problem from a small open reasoning model, and the answer each gives when its thinking is cut short.

Loading the recorded outputs...

Try this

With 2 answers, compare p = 60% with p = 40% as n grows.
With 4 answers, keep p below 50% and compare spread-out wrong answers with one tempting wrong answer.
Run 1,000 votes and check the simulation against the exact curve.
In the recorded traces, lower the thinking budget for a problem and watch which wrong answers appear, then compare real votes with the formula.

The exact voting probabilities in codePython with the standard library

from itertools import product
from math import comb, factorial, prod

def majority(p, n):
    # Two answers, odd n: the vote is right when more than half the samples are.
    return sum(comb(n, k) * p**k * (1 - p) ** (n - k) for k in range(n // 2 + 1, n + 1))

for p in (0.6, 0.4):
    print(f"2 answers, p = {p}:", ", ".join(f"n={n}: {majority(p, n):.3f}" for n in (1, 5, 25)))

def plurality(probs, n):
    # Sum over every tally of n answers (answer 0 is correct); split ties evenly.
    total = 0.0
    for counts in product(range(n + 1), repeat=len(probs) - 1):
        if sum(counts) > n:
            continue
        counts = (*counts, n - sum(counts))
        top = max(counts)
        if counts[0] == top:
            ways = factorial(n) // prod(factorial(c) for c in counts)
            total += ways * prod(q**c for q, c in zip(probs, counts)) / counts.count(top)
    return total

# The demo's four-answer setups. Answer 0 is correct.
spread = [0.4, 0.2, 0.2, 0.2]  # wrong answers spread out
tempting = [0.3, 0.49, 0.105, 0.105]  # one tempting wrong answer
for name, probs in (("4, spread", spread), ("4, one tempting", tempting)):
    accuracy = (f"n={n}: {plurality(probs, n):.3f}" for n in (1, 9, 41))
    print(f"{name}, p = {probs[0]}:", ", ".join(accuracy))

2 answers, p = 0.6: n=1: 0.600, n=5: 0.683, n=25: 0.846
2 answers, p = 0.4: n=1: 0.400, n=5: 0.317, n=25: 0.154
4, spread, p = 0.4: n=1: 0.400, n=9: 0.599, n=41: 0.894
4, one tempting, p = 0.3: n=1: 0.300, n=9: 0.244, n=41: 0.083

The calculator’s curve assumes independent samples. Real samples from one model make correlated mistakes, so real gains are smaller; the recorded traces from DeepSeek-R1-Distill-Qwen-1.5B show by how much.