Shipping and measuring

An offline eval says the new version is better on the examples you collected. Shipping it says whether that is true of the traffic you actually have, which is a different and larger set, and whether the things you did not measure got worse. This section is about the gap between those two statements: why the offline number does not settle it, how to roll out so a mistake is small and reversible, choosing the metric before you can see which one flatters you, and the guardrails that catch the damage a success metric will happily ignore.

Why offline evals are not enough#

Your eval set was assembled by people, and real traffic was not. The gap shows up in four specific ways, and all four are invisible offline by construction.

what offline misses              why
the input distribution           users ask things nobody thought to write down
the long tail                    rare cases are rare in a sample of 300, and common in a month
interaction effects              latency, truncation and retries change the answer
what people do with it           a better answer nobody clicks is not better

None of this makes offline evals optional. They are the cheap filter that stops you shipping something clearly worse, and they are the only feedback fast enough to iterate against. The mistake is treating a passed eval as a decision rather than as permission to try it on a small number of real users.

Rolling out safely#

The rollout question people get wrong is not whether to canary but how big and for how long, and that is arithmetic rather than taste. Here the new version doubles the error rate, from 2 per cent to 4 — a serious regression, not a subtle one — on a service taking five thousand requests an hour.

import numpy as np
rng = np.random.default_rng(0)

def detect(share, hours, rate_per_hour=5000, base=0.02, lift=2.0, trials=3000):
    """A canary on that share of traffic, against the rest as control.
    How often is the regression caught within the window, at 99% confidence?"""
    n = int(rate_per_hour * hours * share)
    hits = 0
    for _ in range(trials):
        control = rng.random(int(rate_per_hour * hours * (1 - share))) < base
        canary = rng.random(n) < base * lift
        pc, pk = control.mean(), canary.mean()
        se = np.sqrt(pc * (1 - pc) / len(control) + pk * (1 - pk) / n)
        if se > 0 and (pk - pc) / se > 2.58:
            hits += 1
    return hits / trials

print(f"{'canary':>7} {'1h':>5} {'4h':>5} {'24h':>5}")
for share in (0.01, 0.05, 0.10, 0.50):
    caught = [f"{round(detect(share, h) * 100):4d}%" for h in (1, 4, 24)]
    print(f"{share:>6.0%} " + " ".join(caught))

#  canary    1h    4h   24h
#     1%    0%    4%   87%
#     5%    8%   76%  100%
#    10%   31%   99%  100%
#    50%   95%  100%  100%

So size the canary from what you need to detect rather than from how nervous you feel. A small share is right when a failure is catastrophic and obvious — it will show up in errors and alerts without statistics — and wrong when the failure is a quality regression that only a comparison can reveal. In practice that means two different rollouts: a brief tiny stage that proves nothing is on fire, then a ten to fifty per cent stage held long enough to measure.

• Make the rollback a config change, not a deploy. If reverting takes a build, you will hesitate, and hesitation is the expensive part.
• Hold each stage long enough to cover a full traffic cycle, including the quiet hours when your worst-served users are active.
• Assign by user, not by request, or one person sees both versions and your comparison is measuring nothing.
• Write down what would make you roll back before you start, and the number that triggers it.
• Watch error rate and latency continuously; the quality metric can be read at the end, but these cannot.

Choosing a metric before launch#

Choose it before the data arrives, because afterwards every candidate metric has a value attached and you will find yourself drawn to whichever one looks best. This is not dishonesty; it is what happens to anyone who cares about the result and has a choice of numbers. The defence is to remove the choice in advance.

One primary metric, chosen because it is what the feature is for, with the threshold written down. If the feature is meant to reduce support tickets, the metric is tickets and not the quality score that is easier to move. And name the metrics you are not going to use, because that is what stops the search when the primary one is disappointing — a rollout that reports a secondary metric it did not pre-register is reporting a search, and an experienced reader treats it that way.

Guardrail metrics#

A guardrail is a metric you do not expect to improve and are checking has not got worse. They exist because a success metric can almost always be moved by damaging something else, and the damage lands somewhere the success metric does not look.

success metric moved              what a guardrail catches
answers per session up            cost per session up with it
engagement up                     latency up, and users retrying
resolution rate up                escalations up a day later
quality score up                  refusals up on a minority of inputs
click-through up                  complaints and unsubscribes up

Set guardrails as thresholds rather than as things to look at. Cost per request may rise by no more than ten per cent, p95 latency by no more than fifty milliseconds, error rate not at all — and a breach stops the rollout automatically rather than starting a discussion about whether it matters. The discussion happens when you write the thresholds, which is when nobody is invested in the answer.