Shipping and measuring

An offline eval says the new version is better on the examples you collected. Shipping it says whether that is true of the traffic you actually have, which is a different and larger set, and whether the things you did not measure got worse. This section is about the gap between those two statements: why the offline number does not settle it, how to roll out so a mistake is small and reversible, choosing the metric before you can see which one flatters you, and the guardrails that catch the damage a success metric will happily ignore.

Why offline evals are not enough#

Your eval set was assembled by people, and real traffic was not. The gap shows up in four specific ways, and all four are invisible offline by construction.

what offline misses              why
the input distribution           users ask things nobody thought to write down
the long tail                    rare cases are rare in a sample of 300, and common in a month
interaction effects              latency, truncation and retries change the answer
what people do with it           a better answer nobody clicks is not better

None of this makes offline evals optional. They are the cheap filter that stops you shipping something clearly worse, and they are the only feedback fast enough to iterate against. The mistake is treating a passed eval as a decision rather than as permission to try it on a small number of real users.

Rolling out safely#

The rollout question people get wrong is not whether to canary but how big and for how long, and that is arithmetic rather than taste. Here the new version doubles the error rate, from 2 per cent to 4 — a serious regression, not a subtle one — on a service taking five thousand requests an hour.

import numpy as np
rng = np.random.default_rng(0)

def detect(share, hours, rate_per_hour=5000, base=0.02, lift=2.0, trials=3000):
    """A canary on that share of traffic, against the rest as control.
    How often is the regression caught within the window, at 99% confidence?"""
    n = int(rate_per_hour * hours * share)
    hits = 0
    for _ in range(trials):
        control = rng.random(int(rate_per_hour * hours * (1 - share))) < base
        canary = rng.random(n) < base * lift
        pc, pk = control.mean(), canary.mean()
        se = np.sqrt(pc * (1 - pc) / len(control) + pk * (1 - pk) / n)
        if se > 0 and (pk - pc) / se > 2.58:
            hits += 1
    return hits / trials

print(f"{'canary':>7} {'1h':>5} {'4h':>5} {'24h':>5}")
for share in (0.01, 0.05, 0.10, 0.50):
    caught = [f"{round(detect(share, h) * 100):4d}%" for h in (1, 4, 24)]
    print(f"{share:>6.0%} " + " ".join(caught))

#  canary    1h    4h   24h
#     1%    0%    4%   87%
#     5%    8%   76%  100%
#    10%   31%   99%  100%
#    50%   95%  100%  100%

So size the canary from what you need to detect rather than from how nervous you feel. A small share is right when a failure is catastrophic and obvious — it will show up in errors and alerts without statistics — and wrong when the failure is a quality regression that only a comparison can reveal. In practice that means two different rollouts: a brief tiny stage that proves nothing is on fire, then a ten to fifty per cent stage held long enough to measure.

• Make the rollback a config change, not a deploy. If reverting takes a build, you will hesitate, and hesitation is the expensive part.
• Hold each stage long enough to cover a full traffic cycle, including the quiet hours when your worst-served users are active.
• Assign by user, not by request, or one person sees both versions and your comparison is measuring nothing.
• Write down what would make you roll back before you start, and the number that triggers it.
• Watch error rate and latency continuously; the quality metric can be read at the end, but these cannot.

Choosing a metric before launch#

Choose it before the data arrives, because afterwards every candidate metric has a value attached and you will find yourself drawn to whichever one looks best. This is not dishonesty; it is what happens to anyone who cares about the result and has a choice of numbers. The defence is to remove the choice in advance.

One primary metric, chosen because it is what the feature is for, with the threshold written down. If the feature is meant to reduce support tickets, the metric is tickets and not the quality score that is easier to move. And name the metrics you are not going to use, because that is what stops the search when the primary one is disappointing — a rollout that reports a secondary metric it did not pre-register is reporting a search, and an experienced reader treats it that way.

Guardrail metrics#

A guardrail is a metric you do not expect to improve and are checking has not got worse. They exist because a success metric can almost always be moved by damaging something else, and the damage lands somewhere the success metric does not look.

success metric moved              what a guardrail catches
answers per session up            cost per session up with it
engagement up                     latency up, and users retrying
resolution rate up                escalations up a day later
quality score up                  refusals up on a minority of inputs
click-through up                  complaints and unsubscribes up

Set guardrails as thresholds rather than as things to look at. Cost per request may rise by no more than ten per cent, p95 latency by no more than fifty milliseconds, error rate not at all — and a breach stops the rollout automatically rather than starting a discussion about whether it matters. The discussion happens when you write the thresholds, which is when nobody is invested in the answer.

An online experiment is the only measurement that settles whether a change helped real users, and it is easy to run in a way that produces confident nonsense. Two failures account for most of it: the experiment is too small to detect the effect it was looking for, and somebody watched the dashboard and stopped when it looked good. Both are arithmetic rather than judgement, and this section works both out with numbers you can reproduce.

How many users you need#

The sample size follows from three things you choose before launching: the current conversion rate, the smallest lift worth detecting, and how often you are willing to be wrong in each direction. Run the calculation first, because its usual result is that the experiment you were about to run could not have detected the effect you were hoping for.

import numpy as np

def users_per_arm(baseline, rel_lift, alpha=0.05, power=0.80):
    """1.96 and 0.84 are the z values for a 5% false-positive rate
    and 80% power -- the conventional defaults."""
    p1 = baseline
    p2 = baseline * (1 + rel_lift)
    pbar = (p1 + p2) / 2
    z_a, z_b = 1.959964, 0.841621
    n = (z_a + z_b) ** 2 * 2 * pbar * (1 - pbar) / (p2 - p1) ** 2
    return int(np.ceil(n))

for lift in (0.50, 0.20, 0.10, 0.05, 0.02):
    n = users_per_arm(0.10, lift)
    print(f"{lift:>4.0%} lift  {n:>9,} per arm   {2*n/5000:>5.1f} days at 5,000/day")

#  50% lift        687 per arm     0.3 days at 5,000/day
#  20% lift      3,843 per arm     1.5 days
#  10% lift     14,752 per arm     5.9 days
#   5% lift     57,764 per arm    23.1 days
#   2% lift    356,336 per arm   142.5 days

That table is the honest answer to most A/B testing ambitions at small scale. If you have five thousand users a day, you can measure large effects quickly and small effects never, so the useful move is to stop trying to measure the small ones and make bigger changes instead. And since the formula is easy to get wrong, it is worth checking rather than trusting.

rng = np.random.default_rng(0)

def power_of(n, baseline=0.10, rel=0.10, trials=3000):
    """Simulate the experiment the formula sized, and count how often
    it actually detects a lift that is genuinely there."""
    hits = 0
    p2 = baseline * (1 + rel)
    for _ in range(trials):
        a = rng.random(n) < baseline
        b = rng.random(n) < p2
        xa, xb = a.mean(), b.mean()
        se = np.sqrt(xa * (1 - xa) / n + xb * (1 - xb) / n)
        if se > 0 and (xb - xa) / se > 1.959964:
            hits += 1
    return hits / trials

n = users_per_arm(0.10, 0.10)
print(n, round(power_of(n) * 100, 1))

# 14752 80.1

Why peeking inflates false wins#

The sample size above assumes one test, at the end. Watching a dashboard and stopping the moment it crosses the line is a different procedure with a different error rate, and the difference is not small. Here A and B are identical — the change does nothing at all — and the only variable is how many times the experiment is checked along the way.

def false_wins(peeks, n_final=10_000, p=0.10, trials=4000):
    """A and B are IDENTICAL. How often is a winner declared anyway?"""
    wins = 0
    checkpoints = np.linspace(n_final // peeks, n_final, peeks).astype(int)
    for _ in range(trials):
        a = rng.random(n_final) < p
        b = rng.random(n_final) < p
        for n in checkpoints:
            xa, xb = a[:n].mean(), b[:n].mean()
            se = np.sqrt(xa * (1 - xa) / n + xb * (1 - xb) / n)
            if se > 0 and abs(xa - xb) / se > 1.96:
                wins += 1
                break
    return wins / trials

for peeks in (1, 2, 5, 10, 20, 50):
    print(peeks, round(false_wins(peeks) * 100, 1))

#  1 check    4.9 %   <- the 5% you asked for
#  2 checks   8.4 %
#  5 checks  14.1 %
# 10 checks  20.2 %
# 20 checks  25.3 %
# 50 checks  32.1 %

This is why a team that watches dashboards ships a stream of improvements that never show up in the quarterly numbers. Each one was significant when it was stopped. The defences are all forms of deciding in advance: fix the sample size and look once, or use a sequential test designed for continuous monitoring — a group sequential boundary or an always-valid confidence sequence — which spend the error budget deliberately rather than accidentally. Looking at a dashboard for operational reasons is fine; the rule is only that the stopping decision was made before the data arrived.

Checking the split itself#

Before reading the result, check that the experiment ran. If assignment is fifty-fifty, the arms should contain close to the same number of users, and a difference larger than chance means something is broken — a bot filter applied to one arm, a redirect losing users, an assignment that leaks. This is the sample ratio mismatch check, it takes one line, and it invalidates the result rather than adjusting it.

import math

def srm_p_value(n_a, n_b, expected=0.5):
    """Chi-squared with one degree of freedom, by hand. erfc gives the exact
    tail probability, so this needs no stats library. Below about 0.001,
    the split is broken and the result is not readable."""
    total = n_a + n_b
    exp_a, exp_b = total * expected, total * (1 - expected)
    chi2 = (n_a - exp_a) ** 2 / exp_a + (n_b - exp_b) ** 2 / exp_b
    return round(chi2, 3), math.erfc(math.sqrt(chi2 / 2))

print(srm_p_value(50_000, 50_200))      # a normal wobble
print(srm_p_value(50_000, 51_500))      # something is wrong
print(srm_p_value(50_000, 50_000 + 1))  # sanity: a perfect-ish split

# (0.399, 0.5275)         fine, read the result
# (22.167, 2.51e-06)      investigate before reading anything else
# (0.0, 0.9975)           as expected

The unit of randomisation#

Randomise by user, not by request. A user who sees both variants during one session is in neither arm cleanly, and requests from the same user are correlated, so treating them as independent samples understates the variance and produces intervals that are too narrow. The rule is that the unit of randomisation should be the unit you are willing to call independent, and for anything a person experiences that is the person.

Where users interact — a marketplace, a social feed, anything with shared inventory or a shared model of behaviour — even per-user randomisation leaks, because the treatment arm changes what the control arm sees. That is interference, and the usual answers are randomising by region or by time slice instead, both of which cost statistical power in exchange for an unbiased comparison.

Novelty and variance#

Two more effects are worth knowing because they change what the number means rather than how large it is. A visible change gets attention simply for being new, so an effect that is strong in week one and gone by week three was novelty, not improvement — which is an argument for running long enough to see it decay, not just long enough to reach significance. And variance reduction is the cheapest way to buy power: using a pre-experiment measurement of the same metric as a covariate (the technique known as CUPED) commonly cuts the required sample substantially, because much of the variation between users was already visible before the experiment began.