The statistics of an online experiment

An online experiment is the only measurement that settles whether a change helped real users, and it is easy to run in a way that produces confident nonsense. Two failures account for most of it: the experiment is too small to detect the effect it was looking for, and somebody watched the dashboard and stopped when it looked good. Both are arithmetic rather than judgement, and this section works both out with numbers you can reproduce.

How many users you need#

The sample size follows from three things you choose before launching: the current conversion rate, the smallest lift worth detecting, and how often you are willing to be wrong in each direction. Run the calculation first, because its usual result is that the experiment you were about to run could not have detected the effect you were hoping for.

import numpy as np

def users_per_arm(baseline, rel_lift, alpha=0.05, power=0.80):
    """1.96 and 0.84 are the z values for a 5% false-positive rate
    and 80% power -- the conventional defaults."""
    p1 = baseline
    p2 = baseline * (1 + rel_lift)
    pbar = (p1 + p2) / 2
    z_a, z_b = 1.959964, 0.841621
    n = (z_a + z_b) ** 2 * 2 * pbar * (1 - pbar) / (p2 - p1) ** 2
    return int(np.ceil(n))

for lift in (0.50, 0.20, 0.10, 0.05, 0.02):
    n = users_per_arm(0.10, lift)
    print(f"{lift:>4.0%} lift  {n:>9,} per arm   {2*n/5000:>5.1f} days at 5,000/day")

#  50% lift        687 per arm     0.3 days at 5,000/day
#  20% lift      3,843 per arm     1.5 days
#  10% lift     14,752 per arm     5.9 days
#   5% lift     57,764 per arm    23.1 days
#   2% lift    356,336 per arm   142.5 days

That table is the honest answer to most A/B testing ambitions at small scale. If you have five thousand users a day, you can measure large effects quickly and small effects never, so the useful move is to stop trying to measure the small ones and make bigger changes instead. And since the formula is easy to get wrong, it is worth checking rather than trusting.

rng = np.random.default_rng(0)

def power_of(n, baseline=0.10, rel=0.10, trials=3000):
    """Simulate the experiment the formula sized, and count how often
    it actually detects a lift that is genuinely there."""
    hits = 0
    p2 = baseline * (1 + rel)
    for _ in range(trials):
        a = rng.random(n) < baseline
        b = rng.random(n) < p2
        xa, xb = a.mean(), b.mean()
        se = np.sqrt(xa * (1 - xa) / n + xb * (1 - xb) / n)
        if se > 0 and (xb - xa) / se > 1.959964:
            hits += 1
    return hits / trials

n = users_per_arm(0.10, 0.10)
print(n, round(power_of(n) * 100, 1))

# 14752 80.1

Why peeking inflates false wins#

The sample size above assumes one test, at the end. Watching a dashboard and stopping the moment it crosses the line is a different procedure with a different error rate, and the difference is not small. Here A and B are identical — the change does nothing at all — and the only variable is how many times the experiment is checked along the way.

def false_wins(peeks, n_final=10_000, p=0.10, trials=4000):
    """A and B are IDENTICAL. How often is a winner declared anyway?"""
    wins = 0
    checkpoints = np.linspace(n_final // peeks, n_final, peeks).astype(int)
    for _ in range(trials):
        a = rng.random(n_final) < p
        b = rng.random(n_final) < p
        for n in checkpoints:
            xa, xb = a[:n].mean(), b[:n].mean()
            se = np.sqrt(xa * (1 - xa) / n + xb * (1 - xb) / n)
            if se > 0 and abs(xa - xb) / se > 1.96:
                wins += 1
                break
    return wins / trials

for peeks in (1, 2, 5, 10, 20, 50):
    print(peeks, round(false_wins(peeks) * 100, 1))

#  1 check    4.9 %   <- the 5% you asked for
#  2 checks   8.4 %
#  5 checks  14.1 %
# 10 checks  20.2 %
# 20 checks  25.3 %
# 50 checks  32.1 %

This is why a team that watches dashboards ships a stream of improvements that never show up in the quarterly numbers. Each one was significant when it was stopped. The defences are all forms of deciding in advance: fix the sample size and look once, or use a sequential test designed for continuous monitoring — a group sequential boundary or an always-valid confidence sequence — which spend the error budget deliberately rather than accidentally. Looking at a dashboard for operational reasons is fine; the rule is only that the stopping decision was made before the data arrived.

Checking the split itself#

Before reading the result, check that the experiment ran. If assignment is fifty-fifty, the arms should contain close to the same number of users, and a difference larger than chance means something is broken — a bot filter applied to one arm, a redirect losing users, an assignment that leaks. This is the sample ratio mismatch check, it takes one line, and it invalidates the result rather than adjusting it.

import math

def srm_p_value(n_a, n_b, expected=0.5):
    """Chi-squared with one degree of freedom, by hand. erfc gives the exact
    tail probability, so this needs no stats library. Below about 0.001,
    the split is broken and the result is not readable."""
    total = n_a + n_b
    exp_a, exp_b = total * expected, total * (1 - expected)
    chi2 = (n_a - exp_a) ** 2 / exp_a + (n_b - exp_b) ** 2 / exp_b
    return round(chi2, 3), math.erfc(math.sqrt(chi2 / 2))

print(srm_p_value(50_000, 50_200))      # a normal wobble
print(srm_p_value(50_000, 51_500))      # something is wrong
print(srm_p_value(50_000, 50_000 + 1))  # sanity: a perfect-ish split

# (0.399, 0.5275)         fine, read the result
# (22.167, 2.51e-06)      investigate before reading anything else
# (0.0, 0.9975)           as expected

The unit of randomisation#

Randomise by user, not by request. A user who sees both variants during one session is in neither arm cleanly, and requests from the same user are correlated, so treating them as independent samples understates the variance and produces intervals that are too narrow. The rule is that the unit of randomisation should be the unit you are willing to call independent, and for anything a person experiences that is the person.

Where users interact — a marketplace, a social feed, anything with shared inventory or a shared model of behaviour — even per-user randomisation leaks, because the treatment arm changes what the control arm sees. That is interference, and the usual answers are randomising by region or by time slice instead, both of which cost statistical power in exchange for an unbiased comparison.

Novelty and variance#

Two more effects are worth knowing because they change what the number means rather than how large it is. A visible change gets attention simply for being new, so an effect that is strong in week one and gone by week three was novelty, not improvement — which is an argument for running long enough to see it decay, not just long enough to reach significance. And variance reduction is the cheapest way to buy power: using a pre-experiment measurement of the same metric as a covariate (the technique known as CUPED) commonly cuts the required sample substantially, because much of the variation between users was already visible before the experiment began.