Benchmarks & contamination

Benchmark scores are how models get compared and they are weaker evidence than they look. Public benchmarks end up in training data, so a high score may be recall rather than capability. Older ones are saturated, where the remaining gap between models is mostly label noise in the benchmark itself. Many measure something adjacent to what you care about rather than the thing itself. This section is about reading those tables critically, and then building the small private benchmark that actually predicts how a model will do on your work.

A benchmark score is not one thing#

Before any of the failure modes, it helps to know what the number is a number of. A benchmark result is the output of four separate choices: the items, the prompt that wraps them, the sampling settings, and the rule that decides whether a reply counts as correct. Only the first is what people think they are comparing. The fourth is the one that moves the score most and is almost never printed in the table.

Here are eight replies to one multiple-choice question. All eight say B, and B is the right answer. The only thing that varies is the shape of the reply, which is the sort of thing a model changes between versions for reasons that have nothing to do with capability.

import re
# Eight replies to one multiple-choice item. Every one of them says B, which is correct.
OUTPUTS = ["B", "B.", " B ", "(B)", "The answer is B.", "Answer: B",
           "Let me think. A is wrong because the rate is fixed, so the answer must be B.",
           "**B**"]

def exact(t): return t == "B"
def stripped(t): return t.strip().rstrip(".").strip("()*").strip() == "B"
def first_letter(t):
    m = re.search(r"\b([A-D])\b", t)
    return bool(m) and m.group(1) == "B"
def last_letter(t):
    m = re.findall(r"\b([A-D])\b", t)
    return bool(m) and m[-1] == "B"

for name, rule in [("exact string match", exact), ("strip punctuation", stripped),
                   ("first A-D in the text", first_letter), ("last A-D in the text", last_letter)]:
    n = sum(rule(o) for o in OUTPUTS)
    print(f"{name:24s} scores {n}/8 = {n/8*100:5.1f}%")

# exact string match       scores 1/8 =  12.5%
# strip punctuation        scores 5/8 =  62.5%
# first A-D in the text    scores 7/8 =  87.5%
# last A-D in the text     scores 8/8 = 100.0%

Neither end of that range is obviously the right answer, which is the awkward part. Exact match is defensible if the prompt asked for a bare letter and following instructions is part of what you are testing. Taking the last mentioned option is defensible if you want capability separated from formatting, but it also scores the seventh reply correct for a model that reasoned its way there by luck, and it quietly rewards models that ramble. What is not defensible is comparing a number produced by one rule against a number produced by another, which is what happens every time two scores are copied from two papers into one table.

So the first question to ask of any benchmark number is how an answer was extracted and graded, and the first thing to do to your own harness is to run it twice with two grading rules and look at the difference. If the gap between your two models is smaller than the gap between your two graders, you have measured your harness.

Contamination#

Public benchmarks are published on the internet, and the internet is training data. A high score may reflect memorisation rather than capability, and the effect grows as a benchmark ages and gets discussed.

The standard check is n-gram overlap: build the set of every n-word sequence in the training corpus, and flag a test item that shares one with it. Thirteen words is the length most often used. It is worth knowing how much that check actually buys, because it is reported as though it settles the question. Below, a corpus of four thousand documents, and three kinds of test item measured against it — items lifted from the corpus word for word, the same items with three of their eighteen words swapped, and items that have nothing to do with the corpus.

import random
rng = random.Random(0)
C, V = "bcdfghklmnprstvw", "aeiou"
VOCAB = sorted({"".join(rng.choice(C) + rng.choice(V) + rng.choice(C) + rng.choice(V)
                        + rng.choice(C) for _ in range(3)) for _ in range(5000)})
sent = lambda k: " ".join(rng.choice(VOCAB) for _ in range(k))

corpus = [sent(40) for _ in range(4000)]                       # the "training data"
copied = [" ".join(d.split()[:18]) for d in corpus[:100]]      # lifted verbatim
def reword(s, k=3):
    ws = s.split()
    for _ in range(k):
        ws[rng.randrange(len(ws))] = rng.choice(VOCAB)
    return " ".join(ws)
reworded = [reword(s) for s in copied]
fresh = [sent(18) for _ in range(100)]

def ngrams(t, n):
    ws = t.split()
    return {" ".join(ws[i:i + n]) for i in range(len(ws) - n + 1)}

for n in (8, 13):
    index = set()
    for d in corpus:
        index |= ngrams(d, n)
    print(f"{n}-gram overlap against the training corpus")
    for name, items in [("verbatim copies", copied), ("three words changed", reworded),
                        ("genuinely new items", fresh)]:
        print(f"   {name:21s} flagged {sum(bool(ngrams(t, n) & index) for t in items):3d} / 100")

# 8-gram overlap against the training corpus
#    verbatim copies       flagged 100 / 100
#    three words changed   flagged  60 / 100
#    genuinely new items   flagged   0 / 100
# 13-gram overlap against the training corpus
#    verbatim copies       flagged 100 / 100
#    three words changed   flagged  12 / 100
#    genuinely new items   flagged   0 / 100

Read that as what the check can and cannot say. A benchmark that fails an n-gram check is contaminated, and that is useful. A benchmark that passes one has shown only that it was not copied word for word — a question set that was reworded, translated, reformatted into a different multiple-choice layout, or discussed at length on a forum passes cleanly while the model may still have the content. Shortening n to 8 raises the catch rate on reworded items to 60 per cent but starts flagging ordinary coincidences on real text, where common phrases recur; the choice of n is a precision-recall trade like any other, and there is no setting that detects paraphrase.

Which is why the checks that work do not look at the data at all, they look at behaviour. Give the model the first half of a test item and see whether it completes the rest exactly. Compare the score on the original items against the score on the same items reworded by hand. Compare a benchmark published before the model's training cutoff against one published after it. Each of those measures the thing you care about, which is whether the model has seen this, rather than whether these exact words appear in a corpus you were allowed to look at — and for a hosted model you are almost never allowed to look at it, so the behavioural checks are the only ones available.

Saturation, and the tie at the top#

Once models score above roughly 90 percent, remaining differences are mostly noise and label errors in the benchmark itself. Many well-known benchmarks are saturated and no longer distinguish between strong models at all.

Well below saturation, a leaderboard is less informative than it looks for a plainer reason: a score measured on a finite set of questions is an estimate, and the top few entries are usually within each other's uncertainty. The reading-results section of the reading-papers topic does the interval arithmetic — on a thousand-question benchmark, four scores between 69.9 and 74.2 per cent have intervals that all overlap. What is worth adding here is the consequence for the decision you are actually making, which is not 'what is the interval' but 'if I pick the top row, how often have I picked the better model'. Two models whose true accuracies are 72 and 70 per cent, run on the same items:

import numpy as np
from scipy.stats import norm
g = np.random.default_rng(0)
pA, pB = 0.72, 0.70                      # the true accuracies, which we get to know
tA, tB = norm.ppf(pA), norm.ppf(pB)

print("    n   truly better model on top   difference reaches p<0.05")
for n in (100, 500, 1000, 5000, 20_000):
    first = sig = reps = 0
    for _ in range(2000):
        z = g.normal(size=n)                         # item difficulty, shared by both
        a = z + g.normal(0, 0.5, n) < tA
        b = z + g.normal(0, 0.5, n) < tB
        first += a.mean() > b.mean()
        n01, n10 = int((~a & b).sum()), int((a & ~b).sum())
        if n01 + n10:                                # McNemar on the paired results
            sig += (abs(n10 - n01) - 1) ** 2 / (n10 + n01) > 3.841
        reps += 1
    print(f"{n:6d} {first/reps*100:22.1f}% {sig/reps*100:23.1f}%")

#     n   truly better model on top   difference reaches p<0.05
#    100                   61.5%                      3.4%
#    500                   82.2%                     13.5%
#   1000                   91.5%                     24.6%
#   5000                   99.9%                     85.4%
#  20000                  100.0%                    100.0%

Two things fall out of that table. The first is that the ordering is more reliable than the significance test, and the two columns answer different questions: at a thousand items you would be right to bet on the top row, and wrong to report that the difference had been demonstrated. The second is the hundred-item row. A benchmark of a hundred questions ranks two models that differ by two points correctly 61.5 per cent of the time, which is barely better than tossing a coin, and small benchmarks of exactly that size are common. Real leaderboards are worse than this simulation, because the top entries usually differ by less than two points and because the same benchmark is being used to choose among dozens of models rather than two, which multiplies the chances that noise puts the wrong one on top.

Construct validity, and picking one that predicts your task#

Does the benchmark measure what its name claims? A reasoning benchmark of multiple-choice questions may be measuring something closer to pattern recognition. This is the deepest problem and the least discussed, because it cannot be fixed by better data collection.

For choosing a model, though, the question is narrower and answerable: does this benchmark rank models the way my task ranks them? Most public benchmarks agree with each other, because model quality is dominated by one broad axis and almost everything loads on it. That is why they are useful for the first cut and useless for the last one. Your task differs from the average in a specific way — it is in German, or the documents are tables, or the output has to be parseable JSON, or latency matters more than the last two points — and those differences live off the main axis where the benchmarks are least informative.

The way to find out is cheap and almost nobody does it. Take four or five models, run them on thirty tasks of your own, rank them. Rank the same models by each candidate public benchmark. Compare the two rankings — Spearman correlation if you want a number, but a table of two orderings side by side usually settles it by eye. A benchmark that reproduces your ordering has earned the right to shortlist for you next time and saves you running the full set. One that does not is measuring something else, and its position in a press release is not an argument.

• Does the benchmark's task shape match yours — same input length, same output format, same language, same domain?
• Was it published after the training cutoff of the models being compared, or is contamination live?
• Is the metric the one your product cares about, or a proxy chosen because it is easy to score?
• How many items does it have, and is the gap you are reading larger than the table above says it can resolve?
• Does it move when a model you know to be better on your work gets added?

What to do instead#

1. 1.Use public benchmarks to shortlist, never to decide.
2. 2.Build a private set from your real tasks, which cannot be contaminated because it was never published.
3. 3.Include the failures you have already seen in production.
4. 4.Version it, and refresh it as the product changes.

Twenty of your own real tasks tell you more about which model to use than any public leaderboard, and nobody else can train on them.

A public leaderboard answers a question that is not yours, on items that may already be in the model's training data, reported as one number with no error bar. This section builds the alternative: a private benchmark that starts from a decision you actually have to make, written fresh rather than copied from the web, carrying a canary string so you can tell if it leaks. Then the reporting, with several prompt wordings and seeds, because scores move with both, and an interval, because a two-point lead that disappears when you reword the prompt was never a lead.

Start from the decision#

A benchmark exists to answer a question, such as which model to use for contract review or whether a training change improved reasoning. Write the question first, then collect tasks that someone who cared about that question would recognise as the real thing.

How many items#

The usual advice is twenty of your own tasks, and it is good advice in the sense that twenty is enormously better than none. It is worth knowing what twenty can and cannot decide. Two models answer the same items; the table is how often the better one comes out ahead, across twenty thousand simulated benchmark runs at each size, for three different true gaps.

import numpy as np
rng = np.random.default_rng(0)
REPS = 20_000

def picks_right(pA, pB, n, rho=0.6):
    """Both models answer the same n items. rho is how much their errors agree."""
    shared = rng.random((REPS, n))
    dA = rho * shared + (1 - rho) * rng.random((REPS, n))
    dB = rho * shared + (1 - rho) * rng.random((REPS, n))
    a, b = (dA < pA).mean(1), (dB < pB).mean(1)
    return (a > b).mean() + 0.5 * (a == b).mean()

print("true gap        n=20    n=50   n=100   n=200   n=500  n=1000")
for pA, pB in [(0.80, 0.70), (0.80, 0.75), (0.80, 0.78)]:
    row = "  ".join(f"{picks_right(pA, pB, n)*100:5.1f}%" for n in (20, 50, 100, 200, 500, 1000))
    print(f"{pA:.2f} vs {pB:.2f}  {row}")

# true gap        n=20    n=50   n=100   n=200   n=500  n=1000
# 0.80 vs 0.70   87.6%   96.8%   99.6%  100.0%  100.0%  100.0%
# 0.80 vs 0.75   71.0%   81.5%   89.7%   96.5%   99.8%  100.0%
# 0.80 vs 0.78   58.5%   63.8%   69.1%   75.9%   87.0%   94.4%

Read it as a sizing rule rather than a table to memorise. If one model is clearly better on your work — ten points, the kind of gap you can see by reading twenty outputs — then twenty items settles it 88 per cent of the time and there is no reason to build anything larger. If the gap is five points, twenty items is a 71 per cent coin-flip and two hundred gets you to 96.5. If the gap is two points, which is what a leaderboard race looks like, a thousand of your own carefully written items still gets it wrong one time in eighteen, and the honest conclusion is that the two models are interchangeable for your purposes and you should choose on cost, latency or rate limits instead.

That last row is the useful one to carry into a meeting. The question 'which of these two models is better' often has no answer worth the price of finding it, and the work of building a five-hundred item benchmark is better spent on the questions that do: whether either model is good enough to ship, and which of your slices they fail on.

Keep it out of training data#

• Do not publish it, or publish only a sample.
• Write new items rather than copying from the web, where they may already be in training data.
• Add a unique canary string to the files so its appearance in a model's output would reveal training exposure.
• Refresh a share of the items periodically.

Check for contamination anyway#

For public benchmarks, look for overlap between test items and training data with n-gram matching where you have the data. Where you do not, test for memorisation: give the model the first half of a test item and see whether it completes it word for word, or compare scores on original items against lightly reworded versions.

Measure the noise#

Scores move with the exact prompt wording, the answer format, the order of examples and the sampling seed. Run a few prompt variants and seeds and report the spread. A two-point lead that disappears when the prompt is reworded is not a lead.

Runs cost money, so the practical question is where to spend them: more prompt wordings, or more seeds per wording. They are not equivalent, because the two sources of movement are different sizes. Rewording a prompt typically shifts a score by a couple of points; resampling at the same prompt shifts it by well under one. Below, a two-hundred item benchmark on a model whose true accuracy is 0.700, with those two effects set to 2.0 and 0.8 points, and the question is how much the number you report bounces around.

N_ITEMS, PROMPT_SD, SEED_SD = 200, 0.020, 0.008

def spread(n_prompts, n_seeds, reps=5000):
    out = []
    for _ in range(reps):
        acc = []
        for pe in rng.normal(0, PROMPT_SD, n_prompts):       # one offset per wording
            for _ in range(n_seeds):
                p = np.clip(0.70 + pe + rng.normal(0, SEED_SD), 0, 1)
                acc.append(rng.binomial(N_ITEMS, p) / N_ITEMS)
        out.append(np.mean(acc))
    return float(np.std(out))

print(f"benchmark of {N_ITEMS} items, true accuracy 0.700")
print("setup                   runs   sd of the reported score")
for label, npr, nse in [("1 prompt, 1 seed", 1, 1), ("1 prompt, 3 seeds", 1, 3),
                        ("3 prompts, 1 seed", 3, 1), ("3 prompts, 2 seeds", 3, 2),
                        ("8 prompts, 1 seed", 8, 1)]:
    print(f"{label:22s} {npr*nse:5d} {spread(npr, nse)*100:20.2f} pts")

# benchmark of 200 items, true accuracy 0.700
# setup                   runs   sd of the reported score
# 1 prompt, 1 seed           1                 3.85 pts
# 1 prompt, 3 seeds          3                 2.82 pts
# 3 prompts, 1 seed          3                 2.22 pts
# 3 prompts, 2 seeds         6                 1.73 pts
# 8 prompts, 1 seed          8                 1.38 pts

The first row is the one to sit with. A single run of a two-hundred item benchmark has a standard deviation of 3.85 points, so two thirds of the time it lands within four points of the truth and one time in twenty it is off by more than seven and a half. That is before anyone has changed a model. A great deal of reported regression and improvement is this row.

A benchmark that survives a model change#

An internal benchmark is only useful if a score from today can be compared with a score from six months ago, and the usual way one dies is that somebody improves it. An item gets reworded, a grader is tightened, a bad question is deleted, and the time series silently becomes two time series. Four decisions in the harness prevent that, and all four are cheap at the start and expensive to retrofit.

1. 1.Give every item a stable id derived from the question, not its position in the file, so reordering or adding items does not renumber the history.
2. 2.Store the raw model output, not only the pass or fail. When you later change the grader you can rescore every historical run rather than rerunning the models, which is the difference between a morning and a bill.
3. 3.Record the model, the prompt id and the seed on every row. A result without those three is not comparable to anything.
4. 4.Tag each item with a slice — the feature, document type or language it exercises. The total is what you report and the slices are what you act on.

import hashlib, json

# Written into the items file itself. If a model ever emits this string, the
# file reached its training data.
CANARY = "internal-eval-canary-2f41ab9c-do-not-train"

def item_id(item):
    """Stable across edits to the wording of everything except the question."""
    return hashlib.sha256(item["question"].encode()).hexdigest()[:12]

items = [{"question": "What is the refund window for an international order?",
          "expect": "5 working days", "slice": "policy"},
         {"question": "Which team owns the billing service?",
          "expect": "Payments", "slice": "ownership"}]

def record(model, prompt_id, seed, item, reply, passed):
    return {"item": item_id(item), "slice": item["slice"], "model": model,
            "prompt": prompt_id, "seed": seed, "reply": reply, "pass": passed}

rows = [record("model-a", "p1", 0, items[0], "Five working days.", True),
        record("model-a", "p1", 0, items[1], "The Payments team.", True),
        record("model-b", "p1", 0, items[0], "Thirty days.", False),
        record("model-b", "p1", 0, items[1], "Payments.", True)]

header = {"canary": CANARY, "items": len(items), "version": "2026-02-11"}
print(json.dumps(header, sort_keys=True))
for model in ("model-a", "model-b"):
    mine = [r for r in rows if r["model"] == model]
    print(model, sum(r["pass"] for r in mine), "/", len(mine))
print(json.dumps(rows[0], sort_keys=True))

# {"canary": "internal-eval-canary-2f41ab9c-do-not-train", "items": 2, "version": "2026-02-11"}
# model-a 2 / 2
# model-b 1 / 2
# {"item": "c9c1c9fb0114", "model": "model-a", "pass": true, "prompt": "p1",
#  "reply": "Five working days.", "seed": 0, "slice": "policy"}

One more rule, which is a policy rather than code: items are added and retired, never edited. If a question turns out to be ambiguous, retire it and write a new one with a new id, so the history of the old id stays honest. And when a new model exposes a failure you had not thought to test, the new items go into a separate slice rather than into the existing set, because otherwise the score drops for a reason that has nothing to do with the models and you will spend a week finding out why.

Report it properly#

Model A   71.2%  (95% CI 67.5 to 74.8, 600 items, mean of 3 prompts x 2 seeds)
Model B   69.8%  (95% CI 66.0 to 73.4)
Paired comparison on the same items: A wins 58, B wins 50, p = 0.5
Conclusion: no reliable difference on this benchmark.

Reading other people's numbers#

• Was the evaluation setup the same for every model, including prompt format and number of examples?
• Is there any error bar or seed variation?
• Could the test set be in the training data?
• Does the benchmark measure what the headline claims?