Building and reporting a benchmark you can trust

A public leaderboard answers a question that is not yours, on items that may already be in the model's training data, reported as one number with no error bar. This section builds the alternative: a private benchmark that starts from a decision you actually have to make, written fresh rather than copied from the web, carrying a canary string so you can tell if it leaks. Then the reporting, with several prompt wordings and seeds, because scores move with both, and an interval, because a two-point lead that disappears when you reword the prompt was never a lead.

Start from the decision#

A benchmark exists to answer a question, such as which model to use for contract review or whether a training change improved reasoning. Write the question first, then collect tasks that someone who cared about that question would recognise as the real thing.

How many items#

The usual advice is twenty of your own tasks, and it is good advice in the sense that twenty is enormously better than none. It is worth knowing what twenty can and cannot decide. Two models answer the same items; the table is how often the better one comes out ahead, across twenty thousand simulated benchmark runs at each size, for three different true gaps.

import numpy as np
rng = np.random.default_rng(0)
REPS = 20_000

def picks_right(pA, pB, n, rho=0.6):
    """Both models answer the same n items. rho is how much their errors agree."""
    shared = rng.random((REPS, n))
    dA = rho * shared + (1 - rho) * rng.random((REPS, n))
    dB = rho * shared + (1 - rho) * rng.random((REPS, n))
    a, b = (dA < pA).mean(1), (dB < pB).mean(1)
    return (a > b).mean() + 0.5 * (a == b).mean()

print("true gap        n=20    n=50   n=100   n=200   n=500  n=1000")
for pA, pB in [(0.80, 0.70), (0.80, 0.75), (0.80, 0.78)]:
    row = "  ".join(f"{picks_right(pA, pB, n)*100:5.1f}%" for n in (20, 50, 100, 200, 500, 1000))
    print(f"{pA:.2f} vs {pB:.2f}  {row}")

# true gap        n=20    n=50   n=100   n=200   n=500  n=1000
# 0.80 vs 0.70   87.6%   96.8%   99.6%  100.0%  100.0%  100.0%
# 0.80 vs 0.75   71.0%   81.5%   89.7%   96.5%   99.8%  100.0%
# 0.80 vs 0.78   58.5%   63.8%   69.1%   75.9%   87.0%   94.4%

Read it as a sizing rule rather than a table to memorise. If one model is clearly better on your work — ten points, the kind of gap you can see by reading twenty outputs — then twenty items settles it 88 per cent of the time and there is no reason to build anything larger. If the gap is five points, twenty items is a 71 per cent coin-flip and two hundred gets you to 96.5. If the gap is two points, which is what a leaderboard race looks like, a thousand of your own carefully written items still gets it wrong one time in eighteen, and the honest conclusion is that the two models are interchangeable for your purposes and you should choose on cost, latency or rate limits instead.

That last row is the useful one to carry into a meeting. The question 'which of these two models is better' often has no answer worth the price of finding it, and the work of building a five-hundred item benchmark is better spent on the questions that do: whether either model is good enough to ship, and which of your slices they fail on.

Keep it out of training data#

• Do not publish it, or publish only a sample.
• Write new items rather than copying from the web, where they may already be in training data.
• Add a unique canary string to the files so its appearance in a model's output would reveal training exposure.
• Refresh a share of the items periodically.

Check for contamination anyway#

For public benchmarks, look for overlap between test items and training data with n-gram matching where you have the data. Where you do not, test for memorisation: give the model the first half of a test item and see whether it completes it word for word, or compare scores on original items against lightly reworded versions.

Measure the noise#

Scores move with the exact prompt wording, the answer format, the order of examples and the sampling seed. Run a few prompt variants and seeds and report the spread. A two-point lead that disappears when the prompt is reworded is not a lead.

Runs cost money, so the practical question is where to spend them: more prompt wordings, or more seeds per wording. They are not equivalent, because the two sources of movement are different sizes. Rewording a prompt typically shifts a score by a couple of points; resampling at the same prompt shifts it by well under one. Below, a two-hundred item benchmark on a model whose true accuracy is 0.700, with those two effects set to 2.0 and 0.8 points, and the question is how much the number you report bounces around.

N_ITEMS, PROMPT_SD, SEED_SD = 200, 0.020, 0.008

def spread(n_prompts, n_seeds, reps=5000):
    out = []
    for _ in range(reps):
        acc = []
        for pe in rng.normal(0, PROMPT_SD, n_prompts):       # one offset per wording
            for _ in range(n_seeds):
                p = np.clip(0.70 + pe + rng.normal(0, SEED_SD), 0, 1)
                acc.append(rng.binomial(N_ITEMS, p) / N_ITEMS)
        out.append(np.mean(acc))
    return float(np.std(out))

print(f"benchmark of {N_ITEMS} items, true accuracy 0.700")
print("setup                   runs   sd of the reported score")
for label, npr, nse in [("1 prompt, 1 seed", 1, 1), ("1 prompt, 3 seeds", 1, 3),
                        ("3 prompts, 1 seed", 3, 1), ("3 prompts, 2 seeds", 3, 2),
                        ("8 prompts, 1 seed", 8, 1)]:
    print(f"{label:22s} {npr*nse:5d} {spread(npr, nse)*100:20.2f} pts")

# benchmark of 200 items, true accuracy 0.700
# setup                   runs   sd of the reported score
# 1 prompt, 1 seed           1                 3.85 pts
# 1 prompt, 3 seeds          3                 2.82 pts
# 3 prompts, 1 seed          3                 2.22 pts
# 3 prompts, 2 seeds         6                 1.73 pts
# 8 prompts, 1 seed          8                 1.38 pts

The first row is the one to sit with. A single run of a two-hundred item benchmark has a standard deviation of 3.85 points, so two thirds of the time it lands within four points of the truth and one time in twenty it is off by more than seven and a half. That is before anyone has changed a model. A great deal of reported regression and improvement is this row.

A benchmark that survives a model change#

An internal benchmark is only useful if a score from today can be compared with a score from six months ago, and the usual way one dies is that somebody improves it. An item gets reworded, a grader is tightened, a bad question is deleted, and the time series silently becomes two time series. Four decisions in the harness prevent that, and all four are cheap at the start and expensive to retrofit.

1. 1.Give every item a stable id derived from the question, not its position in the file, so reordering or adding items does not renumber the history.
2. 2.Store the raw model output, not only the pass or fail. When you later change the grader you can rescore every historical run rather than rerunning the models, which is the difference between a morning and a bill.
3. 3.Record the model, the prompt id and the seed on every row. A result without those three is not comparable to anything.
4. 4.Tag each item with a slice — the feature, document type or language it exercises. The total is what you report and the slices are what you act on.

import hashlib, json

# Written into the items file itself. If a model ever emits this string, the
# file reached its training data.
CANARY = "internal-eval-canary-2f41ab9c-do-not-train"

def item_id(item):
    """Stable across edits to the wording of everything except the question."""
    return hashlib.sha256(item["question"].encode()).hexdigest()[:12]

items = [{"question": "What is the refund window for an international order?",
          "expect": "5 working days", "slice": "policy"},
         {"question": "Which team owns the billing service?",
          "expect": "Payments", "slice": "ownership"}]

def record(model, prompt_id, seed, item, reply, passed):
    return {"item": item_id(item), "slice": item["slice"], "model": model,
            "prompt": prompt_id, "seed": seed, "reply": reply, "pass": passed}

rows = [record("model-a", "p1", 0, items[0], "Five working days.", True),
        record("model-a", "p1", 0, items[1], "The Payments team.", True),
        record("model-b", "p1", 0, items[0], "Thirty days.", False),
        record("model-b", "p1", 0, items[1], "Payments.", True)]

header = {"canary": CANARY, "items": len(items), "version": "2026-02-11"}
print(json.dumps(header, sort_keys=True))
for model in ("model-a", "model-b"):
    mine = [r for r in rows if r["model"] == model]
    print(model, sum(r["pass"] for r in mine), "/", len(mine))
print(json.dumps(rows[0], sort_keys=True))

# {"canary": "internal-eval-canary-2f41ab9c-do-not-train", "items": 2, "version": "2026-02-11"}
# model-a 2 / 2
# model-b 1 / 2
# {"item": "c9c1c9fb0114", "model": "model-a", "pass": true, "prompt": "p1",
#  "reply": "Five working days.", "seed": 0, "slice": "policy"}

One more rule, which is a policy rather than code: items are added and retired, never edited. If a question turns out to be ambiguous, retire it and write a new one with a new id, so the history of the old id stays honest. And when a new model exposes a failure you had not thought to test, the new items go into a separate slice rather than into the existing set, because otherwise the score drops for a reason that has nothing to do with the models and you will spend a week finding out why.

Report it properly#

Model A   71.2%  (95% CI 67.5 to 74.8, 600 items, mean of 3 prompts x 2 seeds)
Model B   69.8%  (95% CI 66.0 to 73.4)
Paired comparison on the same items: A wins 58, B wins 50, p = 0.5
Conclusion: no reliable difference on this benchmark.

Reading other people's numbers#

• Was the evaluation setup the same for every model, including prompt format and number of examples?
• Is there any error bar or seed variation?
• Could the test set be in the training data?
• Does the benchmark measure what the headline claims?