Choosing a model

Choosing a model looks like a research problem and is really a measurement problem. The leaderboards cannot answer it, because they measure a different task on different data from yours, and the differences between the top few are usually smaller than the uncertainty on your own evaluation. This section is about running the comparison that does answer it: what to measure, how many examples you actually need before the answer means anything, the dimensions besides quality that decide it in practice, and what to do about the fact that the model you choose will be retired.

Why benchmarks do not answer your question#

A public benchmark is a real measurement of something, and that something is not your product. Three gaps separate the two, and each is large on its own. The task is different — a score on graduate-level physics questions predicts very little about extracting fields from your invoices. The data is different, and for anything published before a model was trained there is a genuine chance the test set is in the training data, which the model's own report will usually acknowledge if you look. And the prompt is different: benchmark harnesses use their own formatting and few-shot examples, so the number describes a setup you are not running.

The practical consequence is not that benchmarks are useless, but that they are a shortlist rather than a decision. Use them to pick three candidates and to rule out the ones that are clearly a tier below. Then measure the three on your own task, because that measurement is cheap — usually an afternoon — and it is the only one that describes what you are about to ship.

How many examples you need#

Before building an eval set it is worth knowing how large it has to be, because the intuitive answer is far too small. Here two models differ in the realistic way: the newer one fixes some of the older one's mistakes and breaks a few of its correct answers. Both answer the same examples, so the comparison is paired, which is the strongest test available.

import numpy as np
rng = np.random.default_rng(0)

def called_correctly(n, fix, brk, trials=2000, base=0.70):
    """B fixes that fraction of A's errors and breaks that fraction of its
    correct answers. Returns how often n examples declare B the better model."""
    hits = 0
    for _ in range(trials):
        a = rng.random(n) < base
        b = a.copy()
        b[~a & (rng.random(n) < fix)] = True      # fixes
        b[ a & (rng.random(n) < brk)] = False     # regressions
        d = b.astype(int) - a.astype(int)
        sd = d.std(ddof=1)
        if sd == 0:
            hits += d.mean() > 0
            continue
        hits += (d.mean() / (sd / np.sqrt(n))) > 1.96    # paired t
    return hits / trials

for n in (50, 100, 200, 500, 1000):
    print(n, round(called_correctly(n, 0.20, 0.02) * 100, 1))

# a true gap of 4.6 points
#    50   20.5 % of the time
#   100   37.8
#   200   69.6
#   500   97.7
#  1000  100.0

Run it again with a smaller difference and the conclusion sharpens. A model that fixes ten per cent of errors while breaking three per cent of correct answers is genuinely better by 0.9 points, and two thousand examples still only calls it correctly 44 per cent of the time. A difference that small is not measurable at any eval-set size you are going to build, which is the useful thing to know: stop trying to resolve it and choose on cost, latency or something else that is measurable.

Running your own comparison#

The comparison itself is deliberately unglamorous, and the discipline is in holding everything else constant.

1. 1.Take two to five hundred real inputs, including the awkward ones. Sampled production traffic beats anything written from imagination.
2. 2.Fix the prompt, the temperature and the output format, and change only the model between runs.
3. 3.Score with whatever you already trust: exact match where the task allows it, a validated judge where it does not, and human review of the disagreements.
4. 4.Report the paired difference with an interval, not two separate averages.
5. 5.Read the examples where they differ. That set is small, and it tells you more about which to ship than the aggregate does.

Step five is the one people skip and the one that pays. The aggregate says which model is better on average; the disagreements say whether the newer model is better in the way you need, or merely better at the parts of your traffic you care least about.

The dimensions besides quality#

Quality is one column of the decision and rarely the deciding one, because the top models are usually close enough that the eval cannot separate them. The others are measurable in an afternoon and differ by much more than a few points.

dimension            how to measure it            why it decides
cost per request     tokens in and out x rate     scales with every user
latency, p95         your prompts, your region    p50 hides the problem
context window       does your input fit          a hard limit, not a preference
structured output    does it honour a schema      removes a whole class of parsing
tool calling         reliability on your tools    agents live or die on this
rate limits          requests and tokens/min      the launch-day constraint
data handling        retention and training use   sometimes decides it outright
deprecation history  how long models have lasted  how often you will redo this

Models get retired#

Every model you choose will be deprecated, usually with months rather than years of notice, and the code that assumed otherwise is the code that breaks. Treat the model as a configured dependency rather than a fact of the system: one place in the codebase names it, the name is pinned to a specific version rather than a floating alias, and the eval set can be run against a new one on an afternoon's notice.

That last point is the real defence. A team with a two-hundred-example eval set treats a deprecation as a scheduled task: run the eval against the replacement, read the disagreements, adjust the prompt, ship. A team without one treats it as a crisis, because they have no way to tell whether the new model is worse until users tell them. The eval set you build to choose a model is the same artefact that makes changing it routine, which is the argument for building it properly the first time. The mechanics of the change itself are the next section.

Changing the model behind a running product is the routine maintenance of this job, forced on you by deprecations whether or not you wanted the upgrade. It goes wrong in a specific way: the new model is better on average and worse on the handful of things your prompt had been quietly tuned around, so aggregate quality rises while a subset of users see a regression nobody predicted. This section is the procedure that catches that before users do — pinning, shadow running, a staged rollout and a rollback that works — with the shadow harness and the fallback chain written out and run.

Prompts are tuned to a model#

A prompt that works is the product of many small adjustments, most of them made to fix something one particular model did. The instruction repeated twice because it was ignored once, the example added because the format drifted, the sentence forbidding a preamble — each is a patch for a behaviour that the next model may not have. So a prompt is not portable, and the expectation that it transfers unchanged is where migrations go wrong.

The practical consequence is that a migration is two changes, not one: the model and the prompt. Measure the new model on the old prompt first, because that is the change you are actually deploying, then allow yourself to adjust the prompt and measure again. Reporting only the second number hides whether the gain came from the model you are paying for or the afternoon you spent rewriting instructions.

Pinning versions#

pinned      "claude-sonnet-4-5-20250929"   a specific build, frozen
floating    "claude-sonnet-4-5"            moves under you, silently
alias       "latest"                       moves under you, faster

Pinning does not avoid the migration, it schedules it. The pinned build is eventually retired and you move on your own timetable rather than the provider's, which is the whole benefit. Keep the name in one place — configuration, not scattered through the code — so that the change is one line and the rollback is one line, and record which build produced any output you store, because six months later that is the only way to explain a difference between two saved results.

Running both side by side#

The safest comparison uses real traffic without exposing anyone to the candidate: serve every request from the live model, send the same request to the candidate, and log both. Users are unaffected by construction, including when the candidate fails, and after a few hundred requests you have a paired comparison on exactly the distribution you care about.

import random

class FakeModel:
    """Stands in for a provider call so the control flow here is real and runnable."""
    def __init__(self, name, fail_rate=0.0, quality=0.9):
        self.name, self.fail_rate, self.quality = name, fail_rate, quality
    def __call__(self, prompt, rng):
        if rng.random() < self.fail_rate:
            raise RuntimeError(self.name + " unavailable")
        return {"model": self.name, "ok": rng.random() < self.quality}

def shadow(live, candidate, prompt, rng, log):
    """Serve from live. Run the candidate too, and never let it affect the response."""
    result = live(prompt, rng)
    try:
        other = candidate(prompt, rng)
        log.append((result["ok"], other["ok"]))
    except Exception:
        log.append((result["ok"], None))     # a candidate failure must not surface
    return result

rng = random.Random(0)
live = FakeModel("v1", quality=0.90)
cand = FakeModel("v2", quality=0.93, fail_rate=0.05)

log = []
for i in range(500):
    shadow(live, cand, "request " + str(i), rng, log)

both = [(a, b) for a, b in log if b is not None]
print("candidate answered", len(both), "of", len(log))
print("agree", sum(a == b for a, b in both),
      " candidate-only wins", sum((not a) and b for a, b in both),
      " live-only wins", sum(a and (not b) for a, b in both))

# candidate answered 466 of 500
# agree 382  candidate-only wins 52  live-only wins 32

Note the other number. The candidate failed on 34 of 500 requests and not one user saw it, because the shadow path swallows its errors by design. That is the property worth preserving when you write this yourself: a shadow that can break the live response is worse than no shadow at all.

Shadowing costs you a second call for every request, so run it on a sample rather than everything — five or ten per cent for a day is usually several thousand paired observations, which is far more than the two to five hundred the previous section showed you need.

Routing and fallbacks#

Once more than one model is configured, the same machinery gives you resilience. A fallback chain tries models in order and returns the first that answers, which converts a provider outage from an incident into a latency increase. The rule is that the last entry must be the one you are most confident exists — often a smaller or cheaper model, or a different provider entirely.

from collections import Counter

def call_with_fallback(chain, prompt, rng, served):
    """Try each model in order. The last must be the one you trust most to exist."""
    last = None
    for model in chain:
        try:
            out = model(prompt, rng)
            served[model.name] += 1
            return out
        except Exception as e:
            last = e
            served[model.name + " (failed)"] += 1
    raise last

primary    = FakeModel("primary", fail_rate=0.15)
backup     = FakeModel("backup", fail_rate=0.05)
small      = FakeModel("small-model", fail_rate=0.0)

rng, served = random.Random(0), Counter()
for i in range(1000):
    call_with_fallback([primary, backup, small], "q" + str(i), rng, served)
print(dict(served))

# primary 843, primary (failed) 157
# backup  146, backup  (failed)  11
# small-model 11
# 843 + 146 + 11 = every one of the 1000 requests answered

A migration checklist#

1. 1.Run your eval set against the new model on the unchanged prompt, and record both numbers.
2. 2.Read every example where the two disagree. This is the step that finds the regression the aggregate hides.
3. 3.Adjust the prompt if needed, then re-run, and keep both results so the gain can be attributed.
4. 4.Shadow on a sample of real traffic for a day, and compare on that distribution rather than your eval set alone.
5. 5.Roll out to a small percentage of users, watching latency and error rate as well as quality.
6. 6.Keep the old model configured and one line away for at least a week.
7. 7.Remove the old path only once the new one has survived a full traffic cycle, including whatever your weekly peak is.

The step people cut under time pressure is the second one, and it is the one that pays. An aggregate improvement with an unexamined set of regressions is how a migration ships that is better in the mean and worse for the customers who complain loudest — and the disagreement set is usually a few dozen examples, which is an hour of reading.