Why leaderboard numbers mislead

Benchmark scores are how models get compared and they are weaker evidence than they look. Public benchmarks end up in training data, so a high score may be recall rather than capability. Older ones are saturated, where the remaining gap between models is mostly label noise in the benchmark itself. Many measure something adjacent to what you care about rather than the thing itself. This section is about reading those tables critically, and then building the small private benchmark that actually predicts how a model will do on your work.

A benchmark score is not one thing#

Before any of the failure modes, it helps to know what the number is a number of. A benchmark result is the output of four separate choices: the items, the prompt that wraps them, the sampling settings, and the rule that decides whether a reply counts as correct. Only the first is what people think they are comparing. The fourth is the one that moves the score most and is almost never printed in the table.

Here are eight replies to one multiple-choice question. All eight say B, and B is the right answer. The only thing that varies is the shape of the reply, which is the sort of thing a model changes between versions for reasons that have nothing to do with capability.

import re
# Eight replies to one multiple-choice item. Every one of them says B, which is correct.
OUTPUTS = ["B", "B.", " B ", "(B)", "The answer is B.", "Answer: B",
           "Let me think. A is wrong because the rate is fixed, so the answer must be B.",
           "**B**"]

def exact(t): return t == "B"
def stripped(t): return t.strip().rstrip(".").strip("()*").strip() == "B"
def first_letter(t):
    m = re.search(r"\b([A-D])\b", t)
    return bool(m) and m.group(1) == "B"
def last_letter(t):
    m = re.findall(r"\b([A-D])\b", t)
    return bool(m) and m[-1] == "B"

for name, rule in [("exact string match", exact), ("strip punctuation", stripped),
                   ("first A-D in the text", first_letter), ("last A-D in the text", last_letter)]:
    n = sum(rule(o) for o in OUTPUTS)
    print(f"{name:24s} scores {n}/8 = {n/8*100:5.1f}%")

# exact string match       scores 1/8 =  12.5%
# strip punctuation        scores 5/8 =  62.5%
# first A-D in the text    scores 7/8 =  87.5%
# last A-D in the text     scores 8/8 = 100.0%

Neither end of that range is obviously the right answer, which is the awkward part. Exact match is defensible if the prompt asked for a bare letter and following instructions is part of what you are testing. Taking the last mentioned option is defensible if you want capability separated from formatting, but it also scores the seventh reply correct for a model that reasoned its way there by luck, and it quietly rewards models that ramble. What is not defensible is comparing a number produced by one rule against a number produced by another, which is what happens every time two scores are copied from two papers into one table.

So the first question to ask of any benchmark number is how an answer was extracted and graded, and the first thing to do to your own harness is to run it twice with two grading rules and look at the difference. If the gap between your two models is smaller than the gap between your two graders, you have measured your harness.

Contamination#

Public benchmarks are published on the internet, and the internet is training data. A high score may reflect memorisation rather than capability, and the effect grows as a benchmark ages and gets discussed.

The standard check is n-gram overlap: build the set of every n-word sequence in the training corpus, and flag a test item that shares one with it. Thirteen words is the length most often used. It is worth knowing how much that check actually buys, because it is reported as though it settles the question. Below, a corpus of four thousand documents, and three kinds of test item measured against it — items lifted from the corpus word for word, the same items with three of their eighteen words swapped, and items that have nothing to do with the corpus.

import random
rng = random.Random(0)
C, V = "bcdfghklmnprstvw", "aeiou"
VOCAB = sorted({"".join(rng.choice(C) + rng.choice(V) + rng.choice(C) + rng.choice(V)
                        + rng.choice(C) for _ in range(3)) for _ in range(5000)})
sent = lambda k: " ".join(rng.choice(VOCAB) for _ in range(k))

corpus = [sent(40) for _ in range(4000)]                       # the "training data"
copied = [" ".join(d.split()[:18]) for d in corpus[:100]]      # lifted verbatim
def reword(s, k=3):
    ws = s.split()
    for _ in range(k):
        ws[rng.randrange(len(ws))] = rng.choice(VOCAB)
    return " ".join(ws)
reworded = [reword(s) for s in copied]
fresh = [sent(18) for _ in range(100)]

def ngrams(t, n):
    ws = t.split()
    return {" ".join(ws[i:i + n]) for i in range(len(ws) - n + 1)}

for n in (8, 13):
    index = set()
    for d in corpus:
        index |= ngrams(d, n)
    print(f"{n}-gram overlap against the training corpus")
    for name, items in [("verbatim copies", copied), ("three words changed", reworded),
                        ("genuinely new items", fresh)]:
        print(f"   {name:21s} flagged {sum(bool(ngrams(t, n) & index) for t in items):3d} / 100")

# 8-gram overlap against the training corpus
#    verbatim copies       flagged 100 / 100
#    three words changed   flagged  60 / 100
#    genuinely new items   flagged   0 / 100
# 13-gram overlap against the training corpus
#    verbatim copies       flagged 100 / 100
#    three words changed   flagged  12 / 100
#    genuinely new items   flagged   0 / 100

Read that as what the check can and cannot say. A benchmark that fails an n-gram check is contaminated, and that is useful. A benchmark that passes one has shown only that it was not copied word for word — a question set that was reworded, translated, reformatted into a different multiple-choice layout, or discussed at length on a forum passes cleanly while the model may still have the content. Shortening n to 8 raises the catch rate on reworded items to 60 per cent but starts flagging ordinary coincidences on real text, where common phrases recur; the choice of n is a precision-recall trade like any other, and there is no setting that detects paraphrase.

Which is why the checks that work do not look at the data at all, they look at behaviour. Give the model the first half of a test item and see whether it completes the rest exactly. Compare the score on the original items against the score on the same items reworded by hand. Compare a benchmark published before the model's training cutoff against one published after it. Each of those measures the thing you care about, which is whether the model has seen this, rather than whether these exact words appear in a corpus you were allowed to look at — and for a hosted model you are almost never allowed to look at it, so the behavioural checks are the only ones available.

Saturation, and the tie at the top#

Once models score above roughly 90 percent, remaining differences are mostly noise and label errors in the benchmark itself. Many well-known benchmarks are saturated and no longer distinguish between strong models at all.

Well below saturation, a leaderboard is less informative than it looks for a plainer reason: a score measured on a finite set of questions is an estimate, and the top few entries are usually within each other's uncertainty. The reading-results section of the reading-papers topic does the interval arithmetic — on a thousand-question benchmark, four scores between 69.9 and 74.2 per cent have intervals that all overlap. What is worth adding here is the consequence for the decision you are actually making, which is not 'what is the interval' but 'if I pick the top row, how often have I picked the better model'. Two models whose true accuracies are 72 and 70 per cent, run on the same items:

import numpy as np
from scipy.stats import norm
g = np.random.default_rng(0)
pA, pB = 0.72, 0.70                      # the true accuracies, which we get to know
tA, tB = norm.ppf(pA), norm.ppf(pB)

print("    n   truly better model on top   difference reaches p<0.05")
for n in (100, 500, 1000, 5000, 20_000):
    first = sig = reps = 0
    for _ in range(2000):
        z = g.normal(size=n)                         # item difficulty, shared by both
        a = z + g.normal(0, 0.5, n) < tA
        b = z + g.normal(0, 0.5, n) < tB
        first += a.mean() > b.mean()
        n01, n10 = int((~a & b).sum()), int((a & ~b).sum())
        if n01 + n10:                                # McNemar on the paired results
            sig += (abs(n10 - n01) - 1) ** 2 / (n10 + n01) > 3.841
        reps += 1
    print(f"{n:6d} {first/reps*100:22.1f}% {sig/reps*100:23.1f}%")

#     n   truly better model on top   difference reaches p<0.05
#    100                   61.5%                      3.4%
#    500                   82.2%                     13.5%
#   1000                   91.5%                     24.6%
#   5000                   99.9%                     85.4%
#  20000                  100.0%                    100.0%

Two things fall out of that table. The first is that the ordering is more reliable than the significance test, and the two columns answer different questions: at a thousand items you would be right to bet on the top row, and wrong to report that the difference had been demonstrated. The second is the hundred-item row. A benchmark of a hundred questions ranks two models that differ by two points correctly 61.5 per cent of the time, which is barely better than tossing a coin, and small benchmarks of exactly that size are common. Real leaderboards are worse than this simulation, because the top entries usually differ by less than two points and because the same benchmark is being used to choose among dozens of models rather than two, which multiplies the chances that noise puts the wrong one on top.

Construct validity, and picking one that predicts your task#

Does the benchmark measure what its name claims? A reasoning benchmark of multiple-choice questions may be measuring something closer to pattern recognition. This is the deepest problem and the least discussed, because it cannot be fixed by better data collection.

For choosing a model, though, the question is narrower and answerable: does this benchmark rank models the way my task ranks them? Most public benchmarks agree with each other, because model quality is dominated by one broad axis and almost everything loads on it. That is why they are useful for the first cut and useless for the last one. Your task differs from the average in a specific way — it is in German, or the documents are tables, or the output has to be parseable JSON, or latency matters more than the last two points — and those differences live off the main axis where the benchmarks are least informative.

The way to find out is cheap and almost nobody does it. Take four or five models, run them on thirty tasks of your own, rank them. Rank the same models by each candidate public benchmark. Compare the two rankings — Spearman correlation if you want a number, but a table of two orderings side by side usually settles it by eye. A benchmark that reproduces your ordering has earned the right to shortlist for you next time and saves you running the full set. One that does not is measuring something else, and its position in a press release is not an argument.

• Does the benchmark's task shape match yours — same input length, same output format, same language, same domain?
• Was it published after the training cutoff of the models being compared, or is contamination live?
• Is the metric the one your product cares about, or a proxy chosen because it is easy to score?
• How many items does it have, and is the gap you are reading larger than the table above says it can resolve?
• Does it move when a model you know to be better on your work gets added?

What to do instead#

1. 1.Use public benchmarks to shortlist, never to decide.
2. 2.Build a private set from your real tasks, which cannot be contaminated because it was never published.
3. 3.Include the failures you have already seen in production.
4. 4.Version it, and refresh it as the product changes.

Twenty of your own real tasks tell you more about which model to use than any public leaderboard, and nobody else can train on them.