Serving a model: batch, online, shadow

Training a model and putting it in front of real traffic are different jobs. This section is the second one: choosing between a nightly batch job and a live endpoint, loading models by a registered version rather than a file path, and moving a new model into production in stages, first shadowing live traffic, then taking a small share of it. Versioning and staged rollout are what let you say which model produced a given prediction, and get back to the previous one in a minute when the answer is bad.

Batch first, if you can#

If predictions can be computed ahead of time, such as nightly churn scores, a batch job that writes scores to a table is simpler, cheaper and easier to monitor than a live service. Reach for an online endpoint only when predictions depend on information available at request time. That advice gets repeated without a number attached to it, and the number is larger than people expect. Below, the same fitted model scores the same rows two ways.

import time
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier

X, y = make_classification(n_samples=50_000, n_features=20, random_state=0)
model = HistGradientBoostingClassifier(max_iter=100, random_state=0).fit(X, y)

big = np.repeat(X, 10, axis=0)                       # 500,000 rows in one call
t = time.perf_counter()
model.predict_proba(big)
batched = time.perf_counter() - t

one = X[:1]                                          # one row, 2000 times
t = time.perf_counter()
for _ in range(2000):
    model.predict_proba(one)
single = (time.perf_counter() - t) / 2000

print(f"batched          {len(big) / batched:12,.0f} rows/s  ({batched / len(big) * 1e6:.2f} us per row)")
print(f"one row at a time{1 / single:12,.0f} rows/s  ({single * 1e3:.2f} ms per row)")
print(f"ratio            {single * len(big) / batched:12,.0f}x more CPU per row")

batched             2,372,903 rows/s  (0.42 us per row)
one row at a time         476 rows/s  (2.10 ms per row)
ratio                   4,981x more CPU per row

Several thousand times the CPU per prediction, for the identical model. Almost none of that is arithmetic: it is the fixed cost of a Python call, a dataframe construction, a validation pass and a memory allocation, paid once per row instead of once per half million. Add HTTP, JSON parsing and a web framework on top and the online path costs more still. Against that, the batch job is one instance for a couple of minutes a night and the online service is at least two instances running permanently so that one can fail.

So the real question is not which is faster, it is what the prediction depends on. Three shapes, and the choice is nearly always forced by that one answer.

shape        prediction depends on          freshness        what it costs
batch        data known by last night        up to 24h stale  minutes of one box
streaming    an event that just happened     seconds          always-on consumers
             (a purchase, a login)                            plus a broker
online       what is in this request         current          always-on replicas,
             (the basket, the typed text)                     sized for peak

The mistake this table prevents is building an online endpoint for a prediction whose inputs all came from the warehouse. A churn score computed from ninety days of behaviour does not change between eleven at night and nine in the morning, and serving it live buys latency you do not need, an availability problem you did not have, and the five-thousand-fold cost above. The reverse mistake is rarer and worse: batching a prediction that depends on the current request, and shipping a system that scores yesterday's basket.

An endpoint#

Two things in the code below are the whole point, and both are about what is inside the saved file rather than what is in the service. The preprocessing is part of the artefact, so serving cannot compute a feature differently from training. And the encoder was fitted with handle_unknown set to ignore, so a plan nobody had seen at training time produces a prediction rather than an exception at three in the morning.

import joblib, pandas as pd
from pathlib import Path
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

rng2 = np.random.default_rng(0)
n = 8000
train = pd.DataFrame({
    "tenure_days": rng2.integers(1, 2000, n),
    "plan": rng2.choice(["free", "pro", "team"], n, p=[.6, .3, .1]),
    "tickets_30d": rng2.poisson(1.2, n),
})
churned = (rng2.random(n) < 1 / (1 + np.exp(-(0.5 * train.tickets_30d
                                              - 0.0015 * train.tenure_days
                                              + (train.plan == "free") * 0.8 - 0.6)))).astype(int)

pipe = Pipeline([
    ("prep", ColumnTransformer([
        ("num", StandardScaler(), ["tenure_days", "tickets_30d"]),
        ("cat", OneHotEncoder(handle_unknown="ignore"), ["plan"]),
    ])),
    ("model", HistGradientBoostingClassifier(max_iter=100, random_state=0)),
])
pipe.fit(train, churned)
Path("models").mkdir(exist_ok=True)
joblib.dump(pipe, "models/churn-v7.joblib")

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()
MODEL_VERSION = "churn-v7"
model = joblib.load("models/churn-v7.joblib")        # loaded once, at startup

class Customer(BaseModel):
    tenure_days: int
    plan: str
    tickets_30d: int

@app.post("/predict")
def predict(c: Customer):
    p = model.predict_proba(pd.DataFrame([c.model_dump()]))[0, 1]
    return {"churn_probability": float(p), "model_version": MODEL_VERSION}

print(predict(Customer(tenure_days=30, plan="free", tickets_30d=4)))
print(predict(Customer(tenure_days=1500, plan="team", tickets_30d=0)))
print(predict(Customer(tenure_days=200, plan="enterprise", tickets_30d=1)))   # unseen plan

{'churn_probability': 0.8438511147806604, 'model_version': 'churn-v7'}
{'churn_probability': 0.1956927139633083, 'model_version': 'churn-v7'}
{'churn_probability': 0.35068347756539, 'model_version': 'churn-v7'}

That third line deserves a second of attention, because it is a decision and not a free win. Ignoring an unknown category means the model scores that customer as though the plan column were blank, which is a guess dressed as an answer. It is usually the right trade for an interactive service, where an exception is worse. It is the wrong trade when the unknown value is a signal you needed, and either way the count of unknowns belongs on the dashboard: a rate that jumps from nought to three per cent overnight is an upstream release, not a new market.

A registry#

The version string above is hard-coded, which is fine for one model and unworkable for five. A registry replaces it with a name and an alias. Below is the whole cycle against a local SQLite backend, so it runs on a laptop: two versions registered, an alias pointed at the newer one, the service loading by alias, and the alias moved back.

import mlflow
from mlflow import MlflowClient
from sklearn.linear_model import LogisticRegression

mlflow.set_tracking_uri("sqlite:///mlflow.db")
client = MlflowClient()
Xr, yr = make_classification(n_samples=2000, n_features=10, random_state=0)

versions = []
for tag, C in [("v1", 0.1), ("v2", 1.0)]:
    with mlflow.start_run(run_name=tag) as run:
        m = LogisticRegression(C=C, max_iter=1000).fit(Xr, yr)
        mlflow.log_params({"C": C})
        mlflow.log_metric("train_acc", m.score(Xr, yr))
        mlflow.sklearn.log_model(m, name="model")
        v = mlflow.register_model(f"runs:/{run.info.run_id}/model", "churn")
        versions.append(v.version)

client.set_registered_model_alias("churn", "champion", versions[-1])
served = mlflow.pyfunc.load_model("models:/churn@champion")
print("champion is version", client.get_model_version_by_alias("churn", "champion").version)
print("prediction", served.predict(Xr[:3]))

client.set_registered_model_alias("churn", "champion", versions[0])      # the rollback
print("after rollback, champion is version",
      client.get_model_version_by_alias("churn", "champion").version)

champion is version 2
prediction [1 0 0]
after rollback, champion is version 1

Services load 'the champion' rather than a file path. That one indirection is what makes rollback a single action: promoting and reverting are both a call that moves an alias, recorded with who did it and when, and neither requires a deploy, a rebuild or an engineer who knows where the file lives. Point the service at models:/churn@champion and the question of which model is serving has exactly one answer at any moment, which is also the answer you return in the response body so that a prediction can be traced back to a run.

The discipline to add on top is that the service must not cache the resolved version for the life of the process, or the alias move does nothing until someone restarts every replica, which is the deploy you were trying to avoid. Re-resolve on an interval, or on a signal, and log the version at every change so the timeline is in one place afterwards.

Rolling out#

1. 1.Shadow: the new model scores live traffic alongside the old one; only the old one's output is used. Compare per request, not only in aggregate.
2. 2.Canary: send a small share of traffic to the new model and watch business and technical metrics.
3. 3.Full rollout, with the old version ready to take traffic back.

The shadow step is where most of the value is, and it is routinely done wrong, by comparing the two models' output distributions and concluding they agree. Here are two genuinely different models scored on the same ten thousand live requests.

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

X, y = make_classification(n_samples=40000, n_features=20, n_informative=10,
                           flip_y=0.05, random_state=0)
Xtr, ytr = X[:30000], y[:30000]
Xlive, ylive = X[30000:], y[30000:]

champion = LogisticRegression(max_iter=1000).fit(Xtr, ytr)
challenger = HistGradientBoostingClassifier(max_iter=200, random_state=0).fit(Xtr, ytr)

pc = champion.predict_proba(Xlive)[:, 1]
pn = challenger.predict_proba(Xlive)[:, 1]
T = 0.5

print(f"champion   AUC {roc_auc_score(ylive, pc):.4f}  flagged {(pc > T).mean():.3f}  mean p {pc.mean():.4f}")
print(f"challenger AUC {roc_auc_score(ylive, pn):.4f}  flagged {(pn > T).mean():.3f}  mean p {pn.mean():.4f}")

disagree = (pc > T) != (pn > T)
only_new = (pn > T) & ~(pc > T)
only_old = (pc > T) & ~(pn > T)
print(f"disagree on {disagree.mean():.2%} of requests")
print(f"  new flags {only_new.sum()} the old missed: {ylive[only_new].mean():.1%} really positive")
print(f"  old flags {only_old.sum()} the new drops:  {ylive[only_old].mean():.1%} really positive")

champion   AUC 0.8145  flagged 0.491  mean p 0.4986
challenger AUC 0.9669  flagged 0.500  mean p 0.5011
disagree on 23.57% of requests
  new flags 1226 the old missed: 93.3% really positive
  old flags 1131 the new drops:  6.2% really positive

The share flagged moves from 0.491 to 0.500 and the mean score from 0.4986 to 0.5011. A shadow report built on distributions says the models agree and the change is safe, and it is looking at a model that disagrees with the incumbent on nearly a quarter of all requests. The two errors cancel: 1,226 cases the new model flags and the old one misses are offset by 1,131 going the other way, and the totals match while almost nothing about the individual decisions does.

Compare per request instead, and the answer arrives immediately. Of the cases only the new model flags, 93.3 per cent are genuinely positive; of the cases only the old model flags, 6.2 per cent are. The challenger is clearly better, which the AUC column also says, and the disagreement set is the evidence a reviewer can actually read. When the split is closer to even — half and half — that is the signal to stop, because the new model is trading one set of errors for another rather than reducing them.

• Log both scores and the request id for every shadowed request, so the comparison can be redone later against outcomes nobody had yet.
• Report the disagreement rate and the outcome rate within each disagreement direction, not the two distributions.
• Shadow long enough to cover a weekend. Traffic mix on a Sunday is not traffic mix on a Tuesday, and the disagreement rate moves with it.
• Run the new model's full serving path in shadow, preprocessing included, not just the model object. Half of what breaks is upstream of predict.

What to skip#

• Skip the online endpoint when the inputs are all from the warehouse. It is the most common over-engineering in production ML.
• Skip building your own registry. The one bundled with whichever tracking tool you already run is enough, and the value is in the alias, not the software.
• Skip the canary when the change is a like-for-like retrain and the shadow disagreement rate is small; go from shadow to full rollout with the alias ready to move back.
• Worth memorising: the three serving shapes and what decides between them, and that shadow comparisons must be per request.