MLOps & model lifecycle

Shipping a model is not the end of the work, because the world it was trained on keeps moving. This section is about what happens afterwards: knowing which version is live and what data produced it, watching for the two ways incoming data can drift away from what the model expects, and deciding in advance what triggers a retrain, so the decision is not made in an incident.

Deploying a model and owning one#

Deploying a model is a finite task: package it, put it behind an endpoint, check the endpoint answers. It ends. Owning one does not end, and the difference is the whole of this topic. An owner can answer five questions at any hour of any day, and the machinery in this section exists so that the answers come from a record rather than from someone's memory.

1. 1.Which model is serving right now, and which run produced it? A version string returned with every prediction, resolvable to a run.
2. 2.What data was it trained on, and can that data still be fetched? An immutable snapshot with a content hash, not a table name.
3. 3.How would I know it had stopped working? A monitor on inputs and outputs that does not wait for labels.
4. 4.What would make me retrain it, decided before anyone is under pressure? A written trigger with a number in it.
5. 5.How fast can I put the previous version back, and has anyone done it this quarter? A registry alias, and a drill.

A team that cannot answer all five has deployed a model and has not taken ownership of one. The gap shows up months later, in the form of a business number moving the wrong way and nobody able to say what changed.

A model is not a build artefact#

Ordinary code behaves the same on day 400 as on day 1. A model degrades, because the world it was trained on moves away from the world it now sees. Nothing errors when this happens, which is what makes it dangerous. A web service that breaks returns a 500 and pages someone; a model that breaks returns a confident number that is wrong, at the same latency as before, with a 200 status code.

Version the model, the data and the code together#

To reproduce a model you need the code, the data, the hyperparameters and the environment. Missing any one of them and a result from three months ago is unrecoverable. The common half-measure is to version only the model file, which buys nothing you actually want: you can serve the old weights again, but you cannot say why they behaved differently, cannot retrain them, and cannot produce a like-for-like comparison with the replacement.

model_v14
  code       git sha a3f21e
  data       s3://datasets/training/2026-03-01/ (hash 9c2b...)
  config     lr 3e-4, batch 64, 12 epochs
  env        image ml-base:2026.03
  metrics    val 0.874, test 0.861

The three parts fail in different ways and it is worth knowing which failure you are buying. A versioned model with no code version means you can serve it and never change it, because nobody can rebuild the preprocessing it expects. A versioned model with no data version means you can rebuild the pipeline and get a different model, because the training table has been overwritten twice since. A versioned model with no environment means it loads on someone's laptop and not on the training box, usually because a serialisation format changed under you. Version all three or accept that the model is frozen: whatever it does now is what it will always do.

The practical form of this is a registry entry, not a document. A registry stores each trained model with that metadata attached and a mutable alias pointing at the one that is live, so promoting a model and rolling one back are both recorded actions rather than someone copying a file onto a server. The platform section covers what that looks like in code.

Two kinds of drift#

• Data drift: the inputs change. New product categories, a different user population, an upstream format change. The relationship may still hold.
• Concept drift: the relationship changes. What predicted fraud last year does not now, because the fraud adapted. More serious and harder to detect.

What to monitor#

Input distribution      has the shape of incoming data moved
Prediction distribution has the balance of outputs shifted
Confidence              are predictions getting less certain
Latency and errors      ordinary service health
Accuracy                where labels eventually arrive
Business metric         the thing the model was meant to improve

The prediction distribution is the cheapest useful signal. If a fraud model flagged 2 percent of transactions for six months and now flags 8, something changed, and you know it today rather than when the labels arrive.

Retraining: schedule or trigger#

There are two policies and the argument between them is usually conducted without numbers. On a schedule is simple and predictable and retrains when nothing changed. On a trigger is efficient and needs the monitoring to work. The way to settle it is to simulate the drift you actually expect and price both. Here is that simulation: a binary classifier launched on month zero, with the true relationship rotating over eighteen months so that the weight on one feature decays away and a feature the model has never used becomes important. Concept drift, in other words, not data drift — the inputs look identical throughout.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(7)
W0 = np.array([1.4, -0.9, 0.6, 0.0, 0.0])          # the true coefficients at launch

def month(t, n=6000, rng=rng):
    """Concept drift: the weight on feature 3 rotates into feature 4 over 18 months."""
    w = W0.copy()
    w[3] = 1.2 * min(t / 18.0, 1.0)
    w[2] = 0.6 * (1 - min(t / 18.0, 1.0))
    X = rng.normal(0, 1, size=(n, 5))
    p = 1 / (1 + np.exp(-(X @ w)))
    return X, (rng.random(n) < p).astype(int)

Xtr, ytr = month(0, 20000)
frozen = LogisticRegression(max_iter=1000).fit(Xtr, ytr)
rolling = LogisticRegression(max_iter=1000).fit(Xtr, ytr)
trigger = LogisticRegression(max_iter=1000).fit(Xtr, ytr)
retrains_r = retrains_t = 0

print("month  frozen  monthly  trigger   retrained")
for t in range(1, 19):
    X, y = month(t)
    a_f = roc_auc_score(y, frozen.predict_proba(X)[:, 1])
    a_r = roc_auc_score(y, rolling.predict_proba(X)[:, 1])
    a_t = roc_auc_score(y, trigger.predict_proba(X)[:, 1])
    fired = a_t < 0.80                                # the trigger: AUC below the floor
    print(f"{t:>5} {a_f:>7.4f} {a_r:>8.4f} {a_t:>8.4f}   {'retrain' if fired else '':>7}")
    rolling = LogisticRegression(max_iter=1000).fit(X, y)
    retrains_r += 1
    if fired:
        trigger = LogisticRegression(max_iter=1000).fit(X, y)
        retrains_t += 1
print(f"retrains: monthly {retrains_r}, trigger {retrains_t}")

month  frozen  monthly  trigger   retrained
    1  0.8406   0.8406   0.8406
    4  0.8346   0.8406   0.8346
    8  0.8187   0.8359   0.8187
   11  0.8012   0.8410   0.8012
   14  0.7977   0.8563   0.7977   retrain
   15  0.7960   0.8654   0.8654
   18  0.7700   0.8649   0.8612
retrains: monthly 18, trigger 1

Three things to take from that table. The frozen model lost 7 points of AUC over eighteen months, 0.8406 to 0.7700, at no point failing in any way a service monitor would see. Monthly retraining held it between 0.83 and 0.87 throughout, at a cost of eighteen retrains. The trigger arm retrained once, in month fourteen, and finished at 0.8612 against monthly retraining's 0.8649 — essentially the same place for one-eighteenth of the training compute.

The cost of the trigger is the thirteen months it spent below the monthly arm before it fired, bottoming out at 0.7977. Whether that matters is a business question, not a modelling one, and it is the question to ask before choosing. The fix, when it does matter, is not to abandon triggers but to set the threshold relative to launch rather than at a round number: a trigger at three points below the launch AUC of 0.8406, so 0.8106, would have fired at month eleven instead of fourteen. An absolute floor of 0.80 is a threshold nobody derived, and it is the most common thing on these dashboards.

• Schedule when retraining is cheap, labels are plentiful and the world moves steadily. It needs no monitoring to work and it cannot be forgotten.
• Trigger when retraining is expensive, needs human review, or the data has a seasonal shape that a schedule would chase.
• Either way, validate the candidate against the incumbent on the same held-out window before promoting. A retrained model is not automatically better, and a retrain during an unusual period can be substantially worse.
• Log the trigger's inputs whether or not it fires, so you can see afterwards how close it came.

Rolling out, and rolling back#

Shadow deployment runs the new model alongside the old on real traffic without serving its output, so you can compare on live data before risking anything. Then a canary on a small share of traffic, then a full rollout. The serving section works through both in code.

Rollback is the half that gets designed last and used first. It has to be one action that a person who did not build the system can perform at three in the morning, which in practice means moving a registry alias back and restarting nothing. Two things break it. The first is state written by the new model that the old one cannot read — a new feature column, a changed score scale that downstream thresholds were retuned against — so rolling the model back leaves the system in a configuration neither version was tested in. The second is never having done it. A rollback path that has not been exercised in a drill is a plan, not a capability.

What to skip#

• Do not build automated retraining before you have monitoring. Automatic retraining on drifted data with no validation gate is a machine for deploying worse models quickly.
• Do not monitor every feature. The drift section shows what happens to a dashboard with fifty statistics on it.
• Do not version the model file alone. It is the cheapest of the three things to store and the least useful on its own.
• Worth memorising: the five ownership questions, the two kinds of drift, and that a schedule needs no monitoring to work while a trigger needs it to work perfectly.

A trained model is a file. Everything that makes it a system somebody can operate sits around it: a registry recording which data and which code produced which model and which one is live, a way of computing features identically in training and at serving time, and a decision about how much of this to build yourself. This section covers those pieces and which ones a small team actually needs. Training and serving skew is the one to take seriously, because the model starts seeing inputs unlike anything it learned from and nothing errors.

Model registry#

A registry stores trained models with their metadata: which data, which code, which metrics, and which stage it is in. Promotion from staging to production becomes a recorded action rather than someone copying a file. The piece that earns its keep is the mutable alias — a name such as champion that points at one version — because it makes promotion and rollback the same one-line operation and it gives the question of what is serving exactly one answer. The serving section works through the calls.

Training and serving skew#

The most common silent failure in production ML: features are computed one way during training and a slightly different way at serving time. The model sees inputs unlike anything it learned from, and performance drops with no error. The reason it keeps happening is structural rather than careless. Training reads a warehouse table with the whole history available and computes features over columns; serving has one request, a key-value store and a different language, so somebody writes the feature a second time. Two implementations of one definition will diverge, and the only question is when.

Training:   average spend over the last 30 days, computed in SQL
Serving:    average spend over the last 30 days, computed in Python
            with a subtly different date boundary

Same name, same intent, different numbers. Silent accuracy loss.

The version everyone actually ships, though, is subtler than a date boundary, and it is worth seeing the arithmetic because it explains why the bug survives testing. Consider a service that normalises its inputs by fitting a scaler on whatever batch it has, instead of loading the scaler fitted during training.

import numpy as np, pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score, accuracy_score

rng = np.random.default_rng(3)

def batch(n, mult=1.0, rng=rng):
    """One batch of requests. 'mult' is an upstream unit change on spend_30d."""
    spend = rng.lognormal(4.0, 0.9, n)
    tenure = rng.normal(500, 180, n)
    tickets = rng.poisson(1.5, n)
    logit = 1.1 * (np.log(spend) - 4.0) - 0.004 * (tenure - 500) + 0.5 * tickets - 1.0
    return (pd.DataFrame({"spend_30d": spend * mult, "tenure": tenure, "tickets": tickets}),
            (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int))

train_X, train_y = batch(40000)
scaler = StandardScaler().fit(train_X)
clf = LogisticRegression(max_iter=1000).fit(scaler.transform(train_X), train_y)

live_X, live_y = batch(20000)
correct = clf.predict_proba(scaler.transform(live_X))[:, 1]
refit = clf.predict_proba(StandardScaler().fit_transform(live_X))[:, 1]
print(f"scaler loaded from training  AUC {roc_auc_score(live_y, correct):.4f}  "
      f"acc {accuracy_score(live_y, correct > 0.5):.4f}")
print(f"scaler refit on the batch    AUC {roc_auc_score(live_y, refit):.4f}  "
      f"acc {accuracy_score(live_y, refit > 0.5):.4f}")

for n in [20000, 512, 64, 1]:
    sub = live_X.iloc[:n]
    good = clf.predict_proba(scaler.transform(sub))[:, 1]
    bad = clf.predict_proba(StandardScaler().fit_transform(sub))[:, 1]
    print(f"batch {n:>6}  mean change in p {np.abs(good - bad).mean():.4f}"
          f"  max {np.abs(good - bad).max():.4f}")

scaler loaded from training  AUC 0.7849  acc 0.7168
scaler refit on the batch    AUC 0.7846  acc 0.7162

batch  20000  mean change in p 0.0052  max 0.0118
batch    512  mean change in p 0.0140  max 0.0845
batch     64  mean change in p 0.0281  max 0.0665
batch      1  mean change in p 0.2538  max 0.2538

This is the shape of the problem in one table. Evaluated offline on a large representative batch, the refit is almost exactly right, because the batch statistics happen to match the training statistics. Every test passes. Online, requests arrive one at a time, and fitting a standard scaler to a single row centres it on itself: every feature becomes zero, and the service returns the model's output at the training mean for every customer it sees. The prediction is not noisy, it is constant, and it is wrong by 0.25 on average. The bug cannot be found by the evaluation that was run, because the evaluation was run in the one regime where it does not appear.

The second family is categorical encoding, and it has a loud version and a silent version that are one careless fix apart. One-hot encoding a serving batch independently produces a different set of columns whenever a category happens not to appear in it, which scikit-learn catches: the feature names should match those that were passed during fit. So somebody reindexes onto the training columns, filling the missing ones with zero, which is correct. Then an upstream release capitalises the channel names, every training column is still present, every one of them is filled with zero, and nothing anywhere raises anything.

CH = ["web", "ios", "android", "partner"]
EFFECT = {"web": 0.0, "ios": 0.6, "android": -0.4, "partner": 1.5}
rng4 = np.random.default_rng(4)

def rows(n, channels=CH, p=(.5, .25, .2, .05)):
    w = np.array(p[:len(channels)], dtype=float)
    ch = rng4.choice(channels, n, p=w / w.sum())
    x = rng4.normal(0, 1, n)
    logit = 0.9 * x + np.array([EFFECT[c] for c in ch]) - 0.5
    return (pd.DataFrame({"x": x, "channel": ch}),
            (rng4.random(n) < 1 / (1 + np.exp(-logit))).astype(int))

train, ytr = rows(30000)
TRAIN_COLS = pd.get_dummies(train, columns=["channel"]).columns.tolist()
clf2 = LogisticRegression(max_iter=1000).fit(pd.get_dummies(train, columns=["channel"]), ytr)

live, yl = rows(4000, channels=["web", "ios", "android"], p=(.55, .27, .18))
try:
    clf2.predict_proba(pd.get_dummies(live, columns=["channel"]))
except ValueError as e:
    print("loud failure:", str(e).splitlines()[0])

renamed = live.copy()
renamed["channel"] = renamed["channel"].str.capitalize()      # an upstream release
def encode(df):
    return pd.get_dummies(df, columns=["channel"]).reindex(columns=TRAIN_COLS, fill_value=0)
good = clf2.predict_proba(encode(live))[:, 1]
bad = clf2.predict_proba(encode(renamed))[:, 1]
print(f"correct  AUC {roc_auc_score(yl, good):.4f}  mean p {good.mean():.4f}")
print(f"renamed  AUC {roc_auc_score(yl, bad):.4f}  mean p {bad.mean():.4f}")
print(f"mean change in p {np.abs(good - bad).mean():.4f}  max {np.abs(good - bad).max():.4f}")

loud failure: The feature names should match those that were passed during fit.
correct  AUC 0.7360  mean p 0.4135
renamed  AUC 0.7212  mean p 0.4819
mean change in p 0.0869  max 0.2012

A point five of a point of AUC and seven points on the mean predicted probability, from a capitalisation. No exception, no log line, no change in latency, and the only monitor that would have seen it is the one watching the prediction distribution. The general lesson is that fill_value=0 on a reindex is a decision to treat unknown as absent, and it should be paired with a count: how many rows had every category column zero. That number is nought in a healthy system and it is the alarm.

Feature stores#

A feature store computes each feature once and serves the same value to training and inference, which removes skew by construction. Concretely it is two stores over one definition: an offline store, usually the warehouse, that can answer what this feature was for this key at this timestamp, and an online store, usually a key-value database, that answers what it is now, in single-digit milliseconds. One piece of code writes both.

The other half of what it buys is point-in-time correctness, which is the quieter and more damaging problem. Building a training set means joining labels to features, and the naive join attaches the feature as it is today to a label from March. If the feature is lifetime order count, the March example carries a value that includes the six months of orders that happened after the thing you are predicting. The model learns from the future, scores beautifully offline and collapses in production, and the symptom is always the same: an offline number nobody can reproduce live.

label:    customer 4471 churned on 2026-03-14

wrong:    SELECT order_count FROM features WHERE customer = 4471
          -> 61, as of today

right:    SELECT order_count FROM feature_history
          WHERE customer = 4471 AND valid_from <= '2026-03-14'
          ORDER BY valid_from DESC LIMIT 1
          -> 23, as it stood on the day

You do not need a feature store to get this right. You need feature history with validity timestamps and an as-of join, which is a table design and roughly twenty lines of SQL. The product sells you the online serving half and the operational plumbing.

What to build and what to buy#

The honest ordering, by value per unit of operational burden, for a team with one or two models in production.

1. 1.One implementation of each feature, imported by both the training job and the service. This is free, removes most skew, and is the thing teams skip.
2. 2.Preprocessing inside the saved model artefact, so the service cannot compute it differently even if it wants to.
3. 3.A registry with an alias, for rollback. A day of work using whatever your tracking tool already ships.
4. 4.Monitoring on inputs and predictions. See the drift section for what to alert on and what to merely record.
5. 5.Feature history with validity timestamps, once you have a label that is not scored at the moment the features are computed.
6. 6.A feature store product, when you have several models sharing features and a latency budget that a warehouse query cannot meet. Not before.

What to skip#

• Skip the feature store until two models share a feature. Before that it is a second copy of your warehouse with an on-call rota attached.
• Skip writing the feature twice and testing that the two agree. Testing that two implementations match is more work than having one, and the test decays.
• Worth memorising: preprocessing belongs inside the artefact, unknown-category counts belong on the dashboard, and a feature joined at today's value is a leak.

A model is trained on one slice of the world and then used on whatever arrives next. Drift is the gap between the two, and this section is how to measure it: keep a reference sample, compare each day's inputs and predictions against it with a statistic such as PSI, and watch the outputs when the true labels will not arrive for weeks. Most of the work is deciding what deserves an alert, because a monitor that fires on every feature is a monitor nobody reads, including on the day it is right.

Compare against a reference#

Drift is always relative, so the first decision is what to compare against. Keep a fixed reference sample — usually the validation data the model was signed off on, or a recent week everyone agrees was healthy — and compare each new day's inputs and predictions with it. The tempting alternative, comparing today against yesterday, sounds more responsive and is worse: a distribution that moves one per cent a day never trips a day-on-day comparison and has moved thirty per cent by the end of the month. Compare against a fixed point, and move the fixed point deliberately, when you retrain.

Population stability index#

PSI bins the reference into equal-sized buckets and measures how much the share in each bucket moved. It is a symmetrised relative entropy, which matters only in that it is an effect size: it answers how much the distribution moved, and it does not care how many rows you have. It works for any numeric feature and for model scores, and its thresholds are conventions to tune, not laws.

import numpy as np

def psi(reference, current, bins=10):
    edges = np.quantile(reference, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    ref = np.histogram(reference, edges)[0] / len(reference)
    cur = np.histogram(current, edges)[0] / len(current)
    ref, cur = np.clip(ref, 1e-6, None), np.clip(cur, 1e-6, None)
    return float(np.sum((cur - ref) * np.log(cur / ref)))

# common rule of thumb: below 0.1 stable, 0.1 to 0.25 worth a look, above 0.25 investigate

A detector that fires#

Reading the definition is not the same as watching it work. Below is a five-feature stream where only two features drift, gradually, and three are stationary. The reference is twenty thousand rows from launch; each day is five thousand rows of what arrived.

rng = np.random.default_rng(0)
FEATURES = ["amount", "age_days", "n_sessions", "latency_ms", "basket_size"]

def sample(n, shift=0.0, rng=rng):
    """Five features. 'shift' moves only amount and n_sessions."""
    return np.column_stack([
        rng.lognormal(3.0 + shift, 0.8, n),          # amount: mean drifts up
        rng.normal(400, 120, n),                     # age_days: stable
        rng.poisson(4 + 6 * shift, n),               # n_sessions: drifts up
        rng.normal(180, 40, n),                      # latency_ms: stable
        rng.normal(3.2, 1.1, n),                     # basket_size: stable
    ])

reference = sample(20000)

print(f"{'day':>4} {'shift':>6} " + " ".join(f"{f:>12}" for f in FEATURES))
for day, shift in [(1, 0.0), (10, 0.05), (20, 0.10), (30, 0.20), (40, 0.35)]:
    cur = sample(5000, shift)
    row = [psi(reference[:, j], cur[:, j]) for j in range(5)]
    print(f"{day:>4} {shift:>6.2f} " + " ".join(f"{v:>12.3f}" for v in row))

 day  shift       amount     age_days   n_sessions   latency_ms  basket_size
   1   0.00        0.004        0.001        0.002        0.001        0.002
  10   0.05        0.010        0.003        0.024        0.001        0.002
  20   0.10        0.022        0.003        0.084        0.001        0.003
  30   0.20        0.064        0.001        0.329        0.002        0.002
  40   0.35        0.169        0.003        0.823        0.003        0.001

That table is what a working detector looks like, and the useful part is the three columns that do nothing. They establish that 0.004 is what stationary looks like at five thousand rows a day, which is the only way to know that n_sessions at 0.084 on day twenty is real rather than sampling. Without stationary features in the same table you have a number with no scale, and the 0.1 and 0.25 rules of thumb become the only thing you have, on a problem where they may be far too loose or far too tight.

Why a p-value is the wrong tool here#

The obvious alternative is a two-sample test: the Kolmogorov-Smirnov test for numeric features, chi-squared for categorical ones. Both work, both are one line, and both are a trap at production sample sizes. Here is the same day-twenty comparison scored both ways.

from scipy.stats import ks_2samp

cur = sample(5000, 0.10)
for j, f in enumerate(FEATURES):
    stat, p = ks_2samp(reference[:, j], cur[:, j])
    print(f"{f:>12}  KS {stat:.3f}  p {p:.2e}  PSI {psi(reference[:, j], cur[:, j]):.3f}")

      amount  KS 0.062  p 7.11e-14  PSI 0.020
    age_days  KS 0.015  p 3.46e-01  PSI 0.002
  n_sessions  KS 0.122  p 4.36e-52  PSI 0.085
  latency_ms  KS 0.010  p 8.51e-01  PSI 0.002
 basket_size  KS 0.018  p 1.52e-01  PSI 0.003

Read the amount row. Its PSI is 0.020, comfortably inside anybody's stable band, and its KS p-value is seven parts in a hundred trillion. The test is not wrong: with twenty thousand reference rows and five thousand current rows, a shift that small genuinely is not chance. That is the problem. A significance test answers whether the difference is real, and at production volumes every difference is real. The question you actually have is whether the difference is large enough to matter, and that is what an effect size such as PSI, or the KS statistic itself rather than its p-value, is for.

So use the statistic and ignore the p-value. The KS statistic column works as well as PSI here and is one import: 0.010 to 0.018 for the stationary features, 0.122 for the one that moved. Where PSI earns its keep is that it is defined on categorical features too, and that the banding convention is widely enough known that a risk team will recognise the number.

When labels arrive late#

For fraud or churn, the true outcome may take weeks. Until then, watch what you can: input drift, the distribution of predicted scores, the share of decisions above the threshold, and proxies such as complaints or manual review overturns. When labels do arrive, compute performance by the week the prediction was made, not the week the label landed, or a single bad week smears across the three months of cohorts that were still open.

The failure input drift cannot see#

Everything so far watches the inputs. There is a common failure that leaves the inputs untouched and the ranking metrics untouched, and quietly destroys the probabilities: the base rate moves. A model trained when ten per cent of cases were positive is serving a population where forty per cent are. Its features still separate the classes exactly as well as they did. Its probabilities are now wrong by a quarter, and every downstream expected-value decision is wrong with them.

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, brier_score_loss

def make(n, base_rate, rng):
    """Three features; only the share of positives changes between calls."""
    y = (rng.random(n) < base_rate).astype(int)
    x = rng.normal(0, 1.0, size=(n, 3)) + y[:, None] * 1.6
    return x, y

def ece(y, p, bins=10):
    """Expected calibration error: the mean gap between predicted and actual, by bin."""
    idx = np.clip(np.digitize(p, np.linspace(0, 1, bins + 1)[1:-1]), 0, bins - 1)
    return float(sum(
        (idx == b).mean() * abs(y[idx == b].mean() - p[idx == b].mean())
        for b in range(bins) if (idx == b).any()))

rng2 = np.random.default_rng(1)
Xtr, ytr = make(40000, 0.10, rng2)
clf = LogisticRegression(max_iter=1000).fit(Xtr, ytr)

print("month  true rate     AUC   Brier     ECE   mean p   flagged   precision")
for month, rate in [(0, 0.10), (2, 0.15), (4, 0.20), (6, 0.30), (8, 0.40)]:
    X, y = make(40000, rate, rng2)
    p = clf.predict_proba(X)[:, 1]
    hit = p > 0.5
    print(f"{month:>5} {rate:>10.2f} {roc_auc_score(y, p):>7.4f} "
          f"{brier_score_loss(y, p):>7.4f} {ece(y, p):>7.4f} {p.mean():>8.4f} "
          f"{hit.mean():>9.3f} {y[hit].mean():>11.3f}")

month  true rate     AUC   Brier     ECE   mean p   flagged   precision
    0       0.10  0.9741  0.0309  0.0027   0.1006     0.087       0.840
    2       0.15  0.9740  0.0410  0.0165   0.1348     0.123       0.892
    4       0.20  0.9760  0.0477  0.0310   0.1662     0.157       0.925
    6       0.30  0.9744  0.0689  0.0677   0.2327     0.229       0.953
    8       0.40  0.9745  0.0867  0.1000   0.2999     0.301       0.968

Every number a normal dashboard carries is telling you the model is fine or improving. AUC does not move at all, because it depends only on the ranking, and the ranking is untouched by a change in how many positives there are; any monotone rescaling of the scores leaves it identical, which is exactly the class of damage happening here. Precision at the 0.5 threshold rises from 0.840 to 0.968, which will be reported as a win. Meanwhile the average predicted probability is 0.2999 against a true rate of 0.3999, ECE has gone from 0.0027 to 0.1000, and Brier score has nearly tripled. Anything sized from the probability — expected fraud loss, how many reviewers to roster, which customers clear a cost threshold — is a quarter low and nobody is looking at the metric that says so.

Two consequences for what you monitor. Put a calibration metric on the dashboard, not only a ranking one: mean predicted probability against realised rate is the cheap version and needs no binning. And notice that the prediction-distribution monitor would have caught this on its own — the share flagged went from 8.7 per cent to 30.1 per cent — which is the argument for watching outputs even when you cannot watch outcomes.

The repair is cheaper than retraining and usually better. Fit one logistic regression from the model's own log-odds to the recent labels, which is Platt scaling and adjusts the intercept without touching what the model learned. On eight thousand labelled rows from the drifted month it moves Brier from 0.0858 to 0.0586 and ECE from 0.1005 to 0.0033, leaving AUC at 0.9755 exactly where it was. Recalibrate first; retrain when the ranking itself has decayed.

Alerts people do not ignore#

The statistics are the easy half. The hard half is that the detector above produces five numbers a day per model, a real system has eighty features, and a dashboard with eighty statistics on it is not monitoring — it is a place where drift goes to be unnoticed. With eighty features and a threshold tuned so a stationary feature trips one day in a hundred, roughly one feature alerts every single day, forever, and the team learns within a fortnight that these alerts mean nothing.

• Alert on a handful of signals, not on every feature: the prediction distribution, calibration against realised outcomes, and the five to ten features the model actually leans on, taken from feature importance.
• Compute the rest and store them without alerting. They are what you read during an investigation, and they are worthless as pages.
• Require persistence. Two consecutive days, or three of five, removes nearly all of the single-day noise and delays a genuine signal by one day.
• Alert on the input-data checks first. Most drift alerts that fire turn out to be an upstream unit change or a broken join, and the data-quality suite catches those with a clearer message.
• Every alert links to a dashboard and a runbook naming what to check first, in order. An alert nobody knows how to action is an alert that gets muted.

What to skip#

• Skip the p-values. At production volume they are all significant; use the effect size.
• Skip drift detection on features the model barely uses. It is real, and it does not matter, and it costs you the credibility of the alerts that do.
• Skip the elaborate multivariate detectors until the per-feature ones are in place and trusted. Most production drift is one feature or one upstream release.
• Worth memorising: the PSI bands, that AUC is blind to calibration drift, and that comparing today against yesterday cannot see a slow trend.

Training a model and putting it in front of real traffic are different jobs. This section is the second one: choosing between a nightly batch job and a live endpoint, loading models by a registered version rather than a file path, and moving a new model into production in stages, first shadowing live traffic, then taking a small share of it. Versioning and staged rollout are what let you say which model produced a given prediction, and get back to the previous one in a minute when the answer is bad.

Batch first, if you can#

If predictions can be computed ahead of time, such as nightly churn scores, a batch job that writes scores to a table is simpler, cheaper and easier to monitor than a live service. Reach for an online endpoint only when predictions depend on information available at request time. That advice gets repeated without a number attached to it, and the number is larger than people expect. Below, the same fitted model scores the same rows two ways.

import time
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier

X, y = make_classification(n_samples=50_000, n_features=20, random_state=0)
model = HistGradientBoostingClassifier(max_iter=100, random_state=0).fit(X, y)

big = np.repeat(X, 10, axis=0)                       # 500,000 rows in one call
t = time.perf_counter()
model.predict_proba(big)
batched = time.perf_counter() - t

one = X[:1]                                          # one row, 2000 times
t = time.perf_counter()
for _ in range(2000):
    model.predict_proba(one)
single = (time.perf_counter() - t) / 2000

print(f"batched          {len(big) / batched:12,.0f} rows/s  ({batched / len(big) * 1e6:.2f} us per row)")
print(f"one row at a time{1 / single:12,.0f} rows/s  ({single * 1e3:.2f} ms per row)")
print(f"ratio            {single * len(big) / batched:12,.0f}x more CPU per row")

batched             2,372,903 rows/s  (0.42 us per row)
one row at a time         476 rows/s  (2.10 ms per row)
ratio                   4,981x more CPU per row

Several thousand times the CPU per prediction, for the identical model. Almost none of that is arithmetic: it is the fixed cost of a Python call, a dataframe construction, a validation pass and a memory allocation, paid once per row instead of once per half million. Add HTTP, JSON parsing and a web framework on top and the online path costs more still. Against that, the batch job is one instance for a couple of minutes a night and the online service is at least two instances running permanently so that one can fail.

So the real question is not which is faster, it is what the prediction depends on. Three shapes, and the choice is nearly always forced by that one answer.

shape        prediction depends on          freshness        what it costs
batch        data known by last night        up to 24h stale  minutes of one box
streaming    an event that just happened     seconds          always-on consumers
             (a purchase, a login)                            plus a broker
online       what is in this request         current          always-on replicas,
             (the basket, the typed text)                     sized for peak

The mistake this table prevents is building an online endpoint for a prediction whose inputs all came from the warehouse. A churn score computed from ninety days of behaviour does not change between eleven at night and nine in the morning, and serving it live buys latency you do not need, an availability problem you did not have, and the five-thousand-fold cost above. The reverse mistake is rarer and worse: batching a prediction that depends on the current request, and shipping a system that scores yesterday's basket.

An endpoint#

Two things in the code below are the whole point, and both are about what is inside the saved file rather than what is in the service. The preprocessing is part of the artefact, so serving cannot compute a feature differently from training. And the encoder was fitted with handle_unknown set to ignore, so a plan nobody had seen at training time produces a prediction rather than an exception at three in the morning.

import joblib, pandas as pd
from pathlib import Path
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

rng2 = np.random.default_rng(0)
n = 8000
train = pd.DataFrame({
    "tenure_days": rng2.integers(1, 2000, n),
    "plan": rng2.choice(["free", "pro", "team"], n, p=[.6, .3, .1]),
    "tickets_30d": rng2.poisson(1.2, n),
})
churned = (rng2.random(n) < 1 / (1 + np.exp(-(0.5 * train.tickets_30d
                                              - 0.0015 * train.tenure_days
                                              + (train.plan == "free") * 0.8 - 0.6)))).astype(int)

pipe = Pipeline([
    ("prep", ColumnTransformer([
        ("num", StandardScaler(), ["tenure_days", "tickets_30d"]),
        ("cat", OneHotEncoder(handle_unknown="ignore"), ["plan"]),
    ])),
    ("model", HistGradientBoostingClassifier(max_iter=100, random_state=0)),
])
pipe.fit(train, churned)
Path("models").mkdir(exist_ok=True)
joblib.dump(pipe, "models/churn-v7.joblib")

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()
MODEL_VERSION = "churn-v7"
model = joblib.load("models/churn-v7.joblib")        # loaded once, at startup

class Customer(BaseModel):
    tenure_days: int
    plan: str
    tickets_30d: int

@app.post("/predict")
def predict(c: Customer):
    p = model.predict_proba(pd.DataFrame([c.model_dump()]))[0, 1]
    return {"churn_probability": float(p), "model_version": MODEL_VERSION}

print(predict(Customer(tenure_days=30, plan="free", tickets_30d=4)))
print(predict(Customer(tenure_days=1500, plan="team", tickets_30d=0)))
print(predict(Customer(tenure_days=200, plan="enterprise", tickets_30d=1)))   # unseen plan

{'churn_probability': 0.8438511147806604, 'model_version': 'churn-v7'}
{'churn_probability': 0.1956927139633083, 'model_version': 'churn-v7'}
{'churn_probability': 0.35068347756539, 'model_version': 'churn-v7'}

That third line deserves a second of attention, because it is a decision and not a free win. Ignoring an unknown category means the model scores that customer as though the plan column were blank, which is a guess dressed as an answer. It is usually the right trade for an interactive service, where an exception is worse. It is the wrong trade when the unknown value is a signal you needed, and either way the count of unknowns belongs on the dashboard: a rate that jumps from nought to three per cent overnight is an upstream release, not a new market.

A registry#

The version string above is hard-coded, which is fine for one model and unworkable for five. A registry replaces it with a name and an alias. Below is the whole cycle against a local SQLite backend, so it runs on a laptop: two versions registered, an alias pointed at the newer one, the service loading by alias, and the alias moved back.

import mlflow
from mlflow import MlflowClient
from sklearn.linear_model import LogisticRegression

mlflow.set_tracking_uri("sqlite:///mlflow.db")
client = MlflowClient()
Xr, yr = make_classification(n_samples=2000, n_features=10, random_state=0)

versions = []
for tag, C in [("v1", 0.1), ("v2", 1.0)]:
    with mlflow.start_run(run_name=tag) as run:
        m = LogisticRegression(C=C, max_iter=1000).fit(Xr, yr)
        mlflow.log_params({"C": C})
        mlflow.log_metric("train_acc", m.score(Xr, yr))
        mlflow.sklearn.log_model(m, name="model")
        v = mlflow.register_model(f"runs:/{run.info.run_id}/model", "churn")
        versions.append(v.version)

client.set_registered_model_alias("churn", "champion", versions[-1])
served = mlflow.pyfunc.load_model("models:/churn@champion")
print("champion is version", client.get_model_version_by_alias("churn", "champion").version)
print("prediction", served.predict(Xr[:3]))

client.set_registered_model_alias("churn", "champion", versions[0])      # the rollback
print("after rollback, champion is version",
      client.get_model_version_by_alias("churn", "champion").version)

champion is version 2
prediction [1 0 0]
after rollback, champion is version 1

Services load 'the champion' rather than a file path. That one indirection is what makes rollback a single action: promoting and reverting are both a call that moves an alias, recorded with who did it and when, and neither requires a deploy, a rebuild or an engineer who knows where the file lives. Point the service at models:/churn@champion and the question of which model is serving has exactly one answer at any moment, which is also the answer you return in the response body so that a prediction can be traced back to a run.

The discipline to add on top is that the service must not cache the resolved version for the life of the process, or the alias move does nothing until someone restarts every replica, which is the deploy you were trying to avoid. Re-resolve on an interval, or on a signal, and log the version at every change so the timeline is in one place afterwards.

Rolling out#

1. 1.Shadow: the new model scores live traffic alongside the old one; only the old one's output is used. Compare per request, not only in aggregate.
2. 2.Canary: send a small share of traffic to the new model and watch business and technical metrics.
3. 3.Full rollout, with the old version ready to take traffic back.

The shadow step is where most of the value is, and it is routinely done wrong, by comparing the two models' output distributions and concluding they agree. Here are two genuinely different models scored on the same ten thousand live requests.

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

X, y = make_classification(n_samples=40000, n_features=20, n_informative=10,
                           flip_y=0.05, random_state=0)
Xtr, ytr = X[:30000], y[:30000]
Xlive, ylive = X[30000:], y[30000:]

champion = LogisticRegression(max_iter=1000).fit(Xtr, ytr)
challenger = HistGradientBoostingClassifier(max_iter=200, random_state=0).fit(Xtr, ytr)

pc = champion.predict_proba(Xlive)[:, 1]
pn = challenger.predict_proba(Xlive)[:, 1]
T = 0.5

print(f"champion   AUC {roc_auc_score(ylive, pc):.4f}  flagged {(pc > T).mean():.3f}  mean p {pc.mean():.4f}")
print(f"challenger AUC {roc_auc_score(ylive, pn):.4f}  flagged {(pn > T).mean():.3f}  mean p {pn.mean():.4f}")

disagree = (pc > T) != (pn > T)
only_new = (pn > T) & ~(pc > T)
only_old = (pc > T) & ~(pn > T)
print(f"disagree on {disagree.mean():.2%} of requests")
print(f"  new flags {only_new.sum()} the old missed: {ylive[only_new].mean():.1%} really positive")
print(f"  old flags {only_old.sum()} the new drops:  {ylive[only_old].mean():.1%} really positive")

champion   AUC 0.8145  flagged 0.491  mean p 0.4986
challenger AUC 0.9669  flagged 0.500  mean p 0.5011
disagree on 23.57% of requests
  new flags 1226 the old missed: 93.3% really positive
  old flags 1131 the new drops:  6.2% really positive

The share flagged moves from 0.491 to 0.500 and the mean score from 0.4986 to 0.5011. A shadow report built on distributions says the models agree and the change is safe, and it is looking at a model that disagrees with the incumbent on nearly a quarter of all requests. The two errors cancel: 1,226 cases the new model flags and the old one misses are offset by 1,131 going the other way, and the totals match while almost nothing about the individual decisions does.

Compare per request instead, and the answer arrives immediately. Of the cases only the new model flags, 93.3 per cent are genuinely positive; of the cases only the old model flags, 6.2 per cent are. The challenger is clearly better, which the AUC column also says, and the disagreement set is the evidence a reviewer can actually read. When the split is closer to even — half and half — that is the signal to stop, because the new model is trading one set of errors for another rather than reducing them.

• Log both scores and the request id for every shadowed request, so the comparison can be redone later against outcomes nobody had yet.
• Report the disagreement rate and the outcome rate within each disagreement direction, not the two distributions.
• Shadow long enough to cover a weekend. Traffic mix on a Sunday is not traffic mix on a Tuesday, and the disagreement rate moves with it.
• Run the new model's full serving path in shadow, preprocessing included, not just the model object. Half of what breaks is upstream of predict.

What to skip#

• Skip the online endpoint when the inputs are all from the warehouse. It is the most common over-engineering in production ML.
• Skip building your own registry. The one bundled with whichever tracking tool you already run is enough, and the value is in the alias, not the software.
• Skip the canary when the change is a like-for-like retrain and the shadow disagreement rate is small; go from shadow to full rollout with the alias ready to move back.
• Worth memorising: the three serving shapes and what decides between them, and that shadow comparisons must be per request.