Operating the service day to day

Shipping the service once is the easy part. This section is the year afterwards: releasing a prompt change with the care a code change gets, since it can alter behaviour just as much and usually gets far less review; running an incident when the provider is degraded rather than down; planning capacity against rate limits you do not control; and alerting on cost per hour, so a loop that retries forever is caught the same afternoon rather than on the monthly invoice.

Prompt changes are deployments#

A prompt edit can change behaviour as much as a code change, and it is often made with far less care. Version prompts, run the eval suite before shipping them, and roll them out gradually like any other change.

Incidents#

1. 1.Is it us or the provider? Check the provider's status first.
2. 2.Did something change? Recent deploys, prompt edits, model version changes.
3. 3.What is the blast radius? All users, one tenant, one feature.
4. 4.Mitigate before diagnosing: roll back, switch to a fallback model, or disable the feature.
5. 5.Then find the cause, and write down what would have caught it earlier.

Capacity and cost#

Plan capacity against your provider rate limits, not your servers. Alert on cost per hour, not just total monthly spend, so a runaway process is caught within the hour rather than at the end of the month.

Shadow, canary, rollback#

Every change worth being careful about goes through the same three stages, and they answer different questions. Shadow answers whether the new thing produces different output, on real traffic, with nobody exposed to it. Canary answers whether the difference is bad, on a small share of real users. Rollback is what you do when the answer is yes, and it is the stage that gets designed last and used first.

The question nobody asks until it is too late is how long the canary has to run. It is not a judgement call; it is a sample size, and the sample size for a rate is set by the baseline and the size of the change you want to notice. Working it out takes one function, and the answer reframes the whole exercise.

import numpy as np
from scipy import stats

def n_needed(baseline, effect, power=0.8, alpha=0.05):
    """Requests per arm to notice a move from 'baseline' to 'baseline + effect'
    in a rate, at 80 per cent power and the usual 5 per cent significance."""
    p1, p2 = baseline, baseline + effect
    pbar = (p1 + p2) / 2
    z_a, z_b = stats.norm.ppf(1 - alpha / 2), stats.norm.ppf(power)
    num = (z_a * np.sqrt(2 * pbar * (1 - pbar))
           + z_b * np.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2
    return int(np.ceil(num / effect ** 2))

CASES = [("error rate", 0.010, 0.005), ("error rate", 0.010, 0.002),
         ("conversion", 0.040, 0.004), ("fraud catch rate", 0.600, 0.030)]
print(f"{'metric':<18} {'base':>6} {'detect':>7} {'n per arm':>11}"
      f" {'5% of 15/s':>12} {'5% of 200/s':>13}")
for name, base, eff in CASES:
    n = n_needed(base, eff)
    print(f"{name:<18} {base:>6.3f} {eff:>7.3f} {n:>11,}"
          f" {n / (15 * 0.05) / 3600:>11.1f}h {n / (200 * 0.05) / 3600:>12.1f}h")

metric               base  detect   n per arm   5% of 15/s   5% of 200/s
error rate          0.010   0.005       7,750         2.9h          0.2h
error rate          0.010   0.002      42,693        15.8h          1.2h
conversion          0.040   0.004      39,475        14.6h          1.1h
fraud catch rate    0.600   0.030       4,129         1.5h          0.1h

A five per cent canary on a fifteen-requests-a-second service needs fifteen hours to notice a two-tenths-of-a-point move in the error rate, and fourteen hours to notice a small conversion change. The half-hour canary that everyone actually runs has nothing to say about either. What it does have something to say about is crashes, timeouts, latency and anything that goes wrong on nearly every request, because those need tens of samples rather than tens of thousands. So be explicit about which question the canary is answering: half an hour proves the thing does not fall over, and it is not evidence about a business metric.

• Size the canary before starting it, from the metric you intend to watch. If the required time is longer than you will wait, either raise the share of traffic or accept that you are only testing for crashes.
• Watch the guardrails continuously and the business metrics on their own timetable. A canary aborted at two minutes on a latency regression is a success, not a failed experiment.
• Route by a stable key, such as user id, rather than per request. A user who gets the old model on one request and the new one on the next sees the system contradict itself.
• Keep the canary at a fixed share while it runs. Ramping while measuring means the sample is drawn from a moving population, and the comparison no longer means what the arithmetic above says it means.

Rollback has one requirement that dominates all the others: it must be one action, performed by someone who did not build the system, at three in the morning, without a deploy. In practice that means the thing being changed is behind an indirection — a registry alias for a model, a versioned record for a prompt, a flag for a code path — so reverting is a value change rather than a release. The test of whether you have it is not whether the plan exists. It is whether anyone has done it this quarter.

The failures that never appear offline#

A class of production failure has the property that no offline evaluation can find it, because the thing that differs is not the model, the data or the code, but the circumstances of the call. Three of them account for most of what actually happens, and it is worth being able to name all three, because each has a different defence.

The first is feature skew: a feature computed one way in training and another way at serving. The MLOps platform section works through two measured cases, including the one where a normalisation refitted per batch is correct on a 20,000-row evaluation and wrong by a quarter of a probability on the online path, where the batch is one row. The defence is structural, not vigilance: one implementation of each feature, and preprocessing saved inside the model artefact.

The second is a stale feature store. The features exist, they are plausible, they are simply older than the request. Below, a model trained on features as of the scoring day is served features that are one to fourteen days old.

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(9)
N, DAYS = 6000, 60
sessions = np.clip(rng.normal(4, 2, (N, DAYS)) + rng.normal(0, 1.5, (N, 1)), 0, None)
sessions = np.cumsum(sessions, axis=1) / np.arange(1, DAYS + 1)     # the slow-moving part
spike = rng.random((N, DAYS)) < 0.03
sessions = sessions + spike * rng.normal(6, 2, (N, DAYS))           # bursts: the fast part
tenure = rng.integers(1, 2000, N)

def outcome(day):
    logit = (0.45 * (sessions[:, day] - sessions[:, day].mean())
             - 0.0008 * (tenure - 1000) - 0.3)
    return (rng.random(N) < 1 / (1 + np.exp(-logit))).astype(int)

train_day = 30
clf = LogisticRegression(max_iter=1000).fit(
    np.column_stack([sessions[:, train_day], tenure]), outcome(train_day))

score_day = 55
y = outcome(score_day)
print(f"{'feature age':>12} {'AUC':>8} {'flagged':>9} {'precision':>11}")
for lag in [0, 1, 2, 3, 7, 14]:
    p = clf.predict_proba(np.column_stack([sessions[:, score_day - lag], tenure]))[:, 1]
    hit = p > 0.5
    print(f"{lag:>10}d  {roc_auc_score(y, p):>8.4f} {hit.mean():>9.3f} {y[hit].mean():>11.3f}")

 feature age      AUC   flagged   precision
         0d    0.7237     0.342       0.654
         1d    0.6943     0.343       0.624
         2d    0.6919     0.343       0.623
         3d    0.6909     0.347       0.620
         7d    0.6940     0.342       0.625
        14d    0.6884     0.346       0.623

Three points of AUC and three points of precision, lost to a cache that is one day behind, with nothing in any log to say so. The shape of the curve is the useful part: the loss is almost entirely between fresh and one day old, and fourteen days is barely worse than one. That is because only the fast-moving part of the feature carries the signal, and it is gone within a day. So the question to ask of any feature is not how stale the store is, it is which component of this feature moves on the timescale of the decision — and that is answerable offline, by exactly the table above, before the pipeline is built.

The third is a dependency upgrade changing preprocessing. A library bumps a default, a tokeniser normalises whitespace differently, an encoder changes its column order, and the model is fed something subtly unlike its training data. No offline evaluation catches this, because the evaluation runs on the new library too, so both sides move together and agree. The defence is a golden fixture: pin a handful of representative rows, hash the preprocessing output, and fail the build when the hash moves.

import hashlib, json
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler

FIXTURE = pd.DataFrame({
    "tenure_days": [1, 30, 365, 1999],
    "plan": ["free", "pro", "team", "free"],
    "tickets_30d": [0, 2, 7, 1],
})

def build(drop=None):
    return ColumnTransformer([
        ("num", StandardScaler(), ["tenure_days", "tickets_30d"]),
        ("cat", OneHotEncoder(handle_unknown="ignore", drop=drop), ["plan"]),
    ])

def fingerprint(prep, df):
    out = np.asarray(prep.fit_transform(df), dtype=float)
    payload = json.dumps({"names": list(prep.get_feature_names_out()),
                          "values": np.round(out, 6).tolist()}, sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()[:16]

GOLDEN = fingerprint(build(), FIXTURE)
print("golden fingerprint", GOLDEN)

after = fingerprint(build(drop="first"), FIXTURE)      # a default changed under you
print("after upgrade     ", after)
print("guard:", "PASS" if after == GOLDEN else "FAIL - preprocessing changed, do not deploy")

golden fingerprint 2f36ba754b4808cc
after upgrade      ade0a912e4f0f389
guard: FAIL - preprocessing changed, do not deploy

Commit the golden hash alongside the code and regenerate it deliberately, as a reviewed change, never by rerunning the generator when the test goes red. The moment someone is allowed to refresh the fixture to make the build pass, the test has been converted into a slower way of doing nothing.

What to skip#

• Skip the long canary for a change that cannot plausibly move a business metric, such as a dependency bump behind a golden-fixture test. Watch the guardrails for ten minutes and go.
• Skip elaborate multi-arm rollouts with two hundred requests a second. There is not enough traffic to resolve two arms, let alone four.
• Skip writing a rollback plan you will not rehearse. An unrehearsed plan and no plan fail identically, and the first one is more expensive because it was believed.
• Worth memorising: canary duration is a sample-size calculation, rollback must be one action without a deploy, and the three offline-invisible failures are skew, staleness and a changed dependency.