Choosing and changing models

Choosing a model looks like a research problem and is really a measurement problem. The leaderboards cannot answer it, because they measure a different task on different data from yours, and the differences between the top few are usually smaller than the uncertainty on your own evaluation. This section is about running the comparison that does answer it: what to measure, how many examples you actually need before the answer means anything, the dimensions besides quality that decide it in practice, and what to do about the fact that the model you choose will be retired.

Why benchmarks do not answer your question#

A public benchmark is a real measurement of something, and that something is not your product. Three gaps separate the two, and each is large on its own. The task is different — a score on graduate-level physics questions predicts very little about extracting fields from your invoices. The data is different, and for anything published before a model was trained there is a genuine chance the test set is in the training data, which the model's own report will usually acknowledge if you look. And the prompt is different: benchmark harnesses use their own formatting and few-shot examples, so the number describes a setup you are not running.

The practical consequence is not that benchmarks are useless, but that they are a shortlist rather than a decision. Use them to pick three candidates and to rule out the ones that are clearly a tier below. Then measure the three on your own task, because that measurement is cheap — usually an afternoon — and it is the only one that describes what you are about to ship.

How many examples you need#

Before building an eval set it is worth knowing how large it has to be, because the intuitive answer is far too small. Here two models differ in the realistic way: the newer one fixes some of the older one's mistakes and breaks a few of its correct answers. Both answer the same examples, so the comparison is paired, which is the strongest test available.

import numpy as np
rng = np.random.default_rng(0)

def called_correctly(n, fix, brk, trials=2000, base=0.70):
    """B fixes that fraction of A's errors and breaks that fraction of its
    correct answers. Returns how often n examples declare B the better model."""
    hits = 0
    for _ in range(trials):
        a = rng.random(n) < base
        b = a.copy()
        b[~a & (rng.random(n) < fix)] = True      # fixes
        b[ a & (rng.random(n) < brk)] = False     # regressions
        d = b.astype(int) - a.astype(int)
        sd = d.std(ddof=1)
        if sd == 0:
            hits += d.mean() > 0
            continue
        hits += (d.mean() / (sd / np.sqrt(n))) > 1.96    # paired t
    return hits / trials

for n in (50, 100, 200, 500, 1000):
    print(n, round(called_correctly(n, 0.20, 0.02) * 100, 1))

# a true gap of 4.6 points
#    50   20.5 % of the time
#   100   37.8
#   200   69.6
#   500   97.7
#  1000  100.0

Run it again with a smaller difference and the conclusion sharpens. A model that fixes ten per cent of errors while breaking three per cent of correct answers is genuinely better by 0.9 points, and two thousand examples still only calls it correctly 44 per cent of the time. A difference that small is not measurable at any eval-set size you are going to build, which is the useful thing to know: stop trying to resolve it and choose on cost, latency or something else that is measurable.

Running your own comparison#

The comparison itself is deliberately unglamorous, and the discipline is in holding everything else constant.

1. 1.Take two to five hundred real inputs, including the awkward ones. Sampled production traffic beats anything written from imagination.
2. 2.Fix the prompt, the temperature and the output format, and change only the model between runs.
3. 3.Score with whatever you already trust: exact match where the task allows it, a validated judge where it does not, and human review of the disagreements.
4. 4.Report the paired difference with an interval, not two separate averages.
5. 5.Read the examples where they differ. That set is small, and it tells you more about which to ship than the aggregate does.

Step five is the one people skip and the one that pays. The aggregate says which model is better on average; the disagreements say whether the newer model is better in the way you need, or merely better at the parts of your traffic you care least about.

The dimensions besides quality#

Quality is one column of the decision and rarely the deciding one, because the top models are usually close enough that the eval cannot separate them. The others are measurable in an afternoon and differ by much more than a few points.

dimension            how to measure it            why it decides
cost per request     tokens in and out x rate     scales with every user
latency, p95         your prompts, your region    p50 hides the problem
context window       does your input fit          a hard limit, not a preference
structured output    does it honour a schema      removes a whole class of parsing
tool calling         reliability on your tools    agents live or die on this
rate limits          requests and tokens/min      the launch-day constraint
data handling        retention and training use   sometimes decides it outright
deprecation history  how long models have lasted  how often you will redo this

Models get retired#

Every model you choose will be deprecated, usually with months rather than years of notice, and the code that assumed otherwise is the code that breaks. Treat the model as a configured dependency rather than a fact of the system: one place in the codebase names it, the name is pinned to a specific version rather than a floating alias, and the eval set can be run against a new one on an afternoon's notice.

That last point is the real defence. A team with a two-hundred-example eval set treats a deprecation as a scheduled task: run the eval against the replacement, read the disagreements, adjust the prompt, ship. A team without one treats it as a crisis, because they have no way to tell whether the new model is worse until users tell them. The eval set you build to choose a model is the same artefact that makes changing it routine, which is the argument for building it properly the first time. The mechanics of the change itself are the next section.