Changing models without breaking things

Changing the model behind a running product is the routine maintenance of this job, forced on you by deprecations whether or not you wanted the upgrade. It goes wrong in a specific way: the new model is better on average and worse on the handful of things your prompt had been quietly tuned around, so aggregate quality rises while a subset of users see a regression nobody predicted. This section is the procedure that catches that before users do — pinning, shadow running, a staged rollout and a rollback that works — with the shadow harness and the fallback chain written out and run.

Prompts are tuned to a model#

A prompt that works is the product of many small adjustments, most of them made to fix something one particular model did. The instruction repeated twice because it was ignored once, the example added because the format drifted, the sentence forbidding a preamble — each is a patch for a behaviour that the next model may not have. So a prompt is not portable, and the expectation that it transfers unchanged is where migrations go wrong.

The practical consequence is that a migration is two changes, not one: the model and the prompt. Measure the new model on the old prompt first, because that is the change you are actually deploying, then allow yourself to adjust the prompt and measure again. Reporting only the second number hides whether the gain came from the model you are paying for or the afternoon you spent rewriting instructions.

Pinning versions#

pinned      "claude-sonnet-4-5-20250929"   a specific build, frozen
floating    "claude-sonnet-4-5"            moves under you, silently
alias       "latest"                       moves under you, faster

Pinning does not avoid the migration, it schedules it. The pinned build is eventually retired and you move on your own timetable rather than the provider's, which is the whole benefit. Keep the name in one place — configuration, not scattered through the code — so that the change is one line and the rollback is one line, and record which build produced any output you store, because six months later that is the only way to explain a difference between two saved results.

Running both side by side#

The safest comparison uses real traffic without exposing anyone to the candidate: serve every request from the live model, send the same request to the candidate, and log both. Users are unaffected by construction, including when the candidate fails, and after a few hundred requests you have a paired comparison on exactly the distribution you care about.

import random

class FakeModel:
    """Stands in for a provider call so the control flow here is real and runnable."""
    def __init__(self, name, fail_rate=0.0, quality=0.9):
        self.name, self.fail_rate, self.quality = name, fail_rate, quality
    def __call__(self, prompt, rng):
        if rng.random() < self.fail_rate:
            raise RuntimeError(self.name + " unavailable")
        return {"model": self.name, "ok": rng.random() < self.quality}

def shadow(live, candidate, prompt, rng, log):
    """Serve from live. Run the candidate too, and never let it affect the response."""
    result = live(prompt, rng)
    try:
        other = candidate(prompt, rng)
        log.append((result["ok"], other["ok"]))
    except Exception:
        log.append((result["ok"], None))     # a candidate failure must not surface
    return result

rng = random.Random(0)
live = FakeModel("v1", quality=0.90)
cand = FakeModel("v2", quality=0.93, fail_rate=0.05)

log = []
for i in range(500):
    shadow(live, cand, "request " + str(i), rng, log)

both = [(a, b) for a, b in log if b is not None]
print("candidate answered", len(both), "of", len(log))
print("agree", sum(a == b for a, b in both),
      " candidate-only wins", sum((not a) and b for a, b in both),
      " live-only wins", sum(a and (not b) for a, b in both))

# candidate answered 466 of 500
# agree 382  candidate-only wins 52  live-only wins 32

Note the other number. The candidate failed on 34 of 500 requests and not one user saw it, because the shadow path swallows its errors by design. That is the property worth preserving when you write this yourself: a shadow that can break the live response is worse than no shadow at all.

Shadowing costs you a second call for every request, so run it on a sample rather than everything — five or ten per cent for a day is usually several thousand paired observations, which is far more than the two to five hundred the previous section showed you need.

Routing and fallbacks#

Once more than one model is configured, the same machinery gives you resilience. A fallback chain tries models in order and returns the first that answers, which converts a provider outage from an incident into a latency increase. The rule is that the last entry must be the one you are most confident exists — often a smaller or cheaper model, or a different provider entirely.

from collections import Counter

def call_with_fallback(chain, prompt, rng, served):
    """Try each model in order. The last must be the one you trust most to exist."""
    last = None
    for model in chain:
        try:
            out = model(prompt, rng)
            served[model.name] += 1
            return out
        except Exception as e:
            last = e
            served[model.name + " (failed)"] += 1
    raise last

primary    = FakeModel("primary", fail_rate=0.15)
backup     = FakeModel("backup", fail_rate=0.05)
small      = FakeModel("small-model", fail_rate=0.0)

rng, served = random.Random(0), Counter()
for i in range(1000):
    call_with_fallback([primary, backup, small], "q" + str(i), rng, served)
print(dict(served))

# primary 843, primary (failed) 157
# backup  146, backup  (failed)  11
# small-model 11
# 843 + 146 + 11 = every one of the 1000 requests answered

A migration checklist#

1. 1.Run your eval set against the new model on the unchanged prompt, and record both numbers.
2. 2.Read every example where the two disagree. This is the step that finds the regression the aggregate hides.
3. 3.Adjust the prompt if needed, then re-run, and keep both results so the gain can be attributed.
4. 4.Shadow on a sample of real traffic for a day, and compare on that distribution rather than your eval set alone.
5. 5.Roll out to a small percentage of users, watching latency and error rate as well as quality.
6. 6.Keep the old model configured and one line away for at least a week.
7. 7.Remove the old path only once the new one has survived a full traffic cycle, including whatever your weekly peak is.

The step people cut under time pressure is the second one, and it is the one that pays. An aggregate improvement with an unexamined set of regressions is how a migration ships that is better in the mean and worse for the customers who complain loudest — and the disagreement set is usually a few dozen examples, which is an hour of reading.