The platform around the model

A trained model is a file. Everything that makes it a system somebody can operate sits around it: a registry recording which data and which code produced which model and which one is live, a way of computing features identically in training and at serving time, and a decision about how much of this to build yourself. This section covers those pieces and which ones a small team actually needs. Training and serving skew is the one to take seriously, because the model starts seeing inputs unlike anything it learned from and nothing errors.

Model registry#

A registry stores trained models with their metadata: which data, which code, which metrics, and which stage it is in. Promotion from staging to production becomes a recorded action rather than someone copying a file. The piece that earns its keep is the mutable alias — a name such as champion that points at one version — because it makes promotion and rollback the same one-line operation and it gives the question of what is serving exactly one answer. The serving section works through the calls.

Training and serving skew#

The most common silent failure in production ML: features are computed one way during training and a slightly different way at serving time. The model sees inputs unlike anything it learned from, and performance drops with no error. The reason it keeps happening is structural rather than careless. Training reads a warehouse table with the whole history available and computes features over columns; serving has one request, a key-value store and a different language, so somebody writes the feature a second time. Two implementations of one definition will diverge, and the only question is when.

Training:   average spend over the last 30 days, computed in SQL
Serving:    average spend over the last 30 days, computed in Python
            with a subtly different date boundary

Same name, same intent, different numbers. Silent accuracy loss.

The version everyone actually ships, though, is subtler than a date boundary, and it is worth seeing the arithmetic because it explains why the bug survives testing. Consider a service that normalises its inputs by fitting a scaler on whatever batch it has, instead of loading the scaler fitted during training.

import numpy as np, pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score, accuracy_score

rng = np.random.default_rng(3)

def batch(n, mult=1.0, rng=rng):
    """One batch of requests. 'mult' is an upstream unit change on spend_30d."""
    spend = rng.lognormal(4.0, 0.9, n)
    tenure = rng.normal(500, 180, n)
    tickets = rng.poisson(1.5, n)
    logit = 1.1 * (np.log(spend) - 4.0) - 0.004 * (tenure - 500) + 0.5 * tickets - 1.0
    return (pd.DataFrame({"spend_30d": spend * mult, "tenure": tenure, "tickets": tickets}),
            (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int))

train_X, train_y = batch(40000)
scaler = StandardScaler().fit(train_X)
clf = LogisticRegression(max_iter=1000).fit(scaler.transform(train_X), train_y)

live_X, live_y = batch(20000)
correct = clf.predict_proba(scaler.transform(live_X))[:, 1]
refit = clf.predict_proba(StandardScaler().fit_transform(live_X))[:, 1]
print(f"scaler loaded from training  AUC {roc_auc_score(live_y, correct):.4f}  "
      f"acc {accuracy_score(live_y, correct > 0.5):.4f}")
print(f"scaler refit on the batch    AUC {roc_auc_score(live_y, refit):.4f}  "
      f"acc {accuracy_score(live_y, refit > 0.5):.4f}")

for n in [20000, 512, 64, 1]:
    sub = live_X.iloc[:n]
    good = clf.predict_proba(scaler.transform(sub))[:, 1]
    bad = clf.predict_proba(StandardScaler().fit_transform(sub))[:, 1]
    print(f"batch {n:>6}  mean change in p {np.abs(good - bad).mean():.4f}"
          f"  max {np.abs(good - bad).max():.4f}")

scaler loaded from training  AUC 0.7849  acc 0.7168
scaler refit on the batch    AUC 0.7846  acc 0.7162

batch  20000  mean change in p 0.0052  max 0.0118
batch    512  mean change in p 0.0140  max 0.0845
batch     64  mean change in p 0.0281  max 0.0665
batch      1  mean change in p 0.2538  max 0.2538

This is the shape of the problem in one table. Evaluated offline on a large representative batch, the refit is almost exactly right, because the batch statistics happen to match the training statistics. Every test passes. Online, requests arrive one at a time, and fitting a standard scaler to a single row centres it on itself: every feature becomes zero, and the service returns the model's output at the training mean for every customer it sees. The prediction is not noisy, it is constant, and it is wrong by 0.25 on average. The bug cannot be found by the evaluation that was run, because the evaluation was run in the one regime where it does not appear.

The second family is categorical encoding, and it has a loud version and a silent version that are one careless fix apart. One-hot encoding a serving batch independently produces a different set of columns whenever a category happens not to appear in it, which scikit-learn catches: the feature names should match those that were passed during fit. So somebody reindexes onto the training columns, filling the missing ones with zero, which is correct. Then an upstream release capitalises the channel names, every training column is still present, every one of them is filled with zero, and nothing anywhere raises anything.

CH = ["web", "ios", "android", "partner"]
EFFECT = {"web": 0.0, "ios": 0.6, "android": -0.4, "partner": 1.5}
rng4 = np.random.default_rng(4)

def rows(n, channels=CH, p=(.5, .25, .2, .05)):
    w = np.array(p[:len(channels)], dtype=float)
    ch = rng4.choice(channels, n, p=w / w.sum())
    x = rng4.normal(0, 1, n)
    logit = 0.9 * x + np.array([EFFECT[c] for c in ch]) - 0.5
    return (pd.DataFrame({"x": x, "channel": ch}),
            (rng4.random(n) < 1 / (1 + np.exp(-logit))).astype(int))

train, ytr = rows(30000)
TRAIN_COLS = pd.get_dummies(train, columns=["channel"]).columns.tolist()
clf2 = LogisticRegression(max_iter=1000).fit(pd.get_dummies(train, columns=["channel"]), ytr)

live, yl = rows(4000, channels=["web", "ios", "android"], p=(.55, .27, .18))
try:
    clf2.predict_proba(pd.get_dummies(live, columns=["channel"]))
except ValueError as e:
    print("loud failure:", str(e).splitlines()[0])

renamed = live.copy()
renamed["channel"] = renamed["channel"].str.capitalize()      # an upstream release
def encode(df):
    return pd.get_dummies(df, columns=["channel"]).reindex(columns=TRAIN_COLS, fill_value=0)
good = clf2.predict_proba(encode(live))[:, 1]
bad = clf2.predict_proba(encode(renamed))[:, 1]
print(f"correct  AUC {roc_auc_score(yl, good):.4f}  mean p {good.mean():.4f}")
print(f"renamed  AUC {roc_auc_score(yl, bad):.4f}  mean p {bad.mean():.4f}")
print(f"mean change in p {np.abs(good - bad).mean():.4f}  max {np.abs(good - bad).max():.4f}")

loud failure: The feature names should match those that were passed during fit.
correct  AUC 0.7360  mean p 0.4135
renamed  AUC 0.7212  mean p 0.4819
mean change in p 0.0869  max 0.2012

A point five of a point of AUC and seven points on the mean predicted probability, from a capitalisation. No exception, no log line, no change in latency, and the only monitor that would have seen it is the one watching the prediction distribution. The general lesson is that fill_value=0 on a reindex is a decision to treat unknown as absent, and it should be paired with a count: how many rows had every category column zero. That number is nought in a healthy system and it is the alarm.

Feature stores#

A feature store computes each feature once and serves the same value to training and inference, which removes skew by construction. Concretely it is two stores over one definition: an offline store, usually the warehouse, that can answer what this feature was for this key at this timestamp, and an online store, usually a key-value database, that answers what it is now, in single-digit milliseconds. One piece of code writes both.

The other half of what it buys is point-in-time correctness, which is the quieter and more damaging problem. Building a training set means joining labels to features, and the naive join attaches the feature as it is today to a label from March. If the feature is lifetime order count, the March example carries a value that includes the six months of orders that happened after the thing you are predicting. The model learns from the future, scores beautifully offline and collapses in production, and the symptom is always the same: an offline number nobody can reproduce live.

label:    customer 4471 churned on 2026-03-14

wrong:    SELECT order_count FROM features WHERE customer = 4471
          -> 61, as of today

right:    SELECT order_count FROM feature_history
          WHERE customer = 4471 AND valid_from <= '2026-03-14'
          ORDER BY valid_from DESC LIMIT 1
          -> 23, as it stood on the day

You do not need a feature store to get this right. You need feature history with validity timestamps and an as-of join, which is a table design and roughly twenty lines of SQL. The product sells you the online serving half and the operational plumbing.

What to build and what to buy#

The honest ordering, by value per unit of operational burden, for a team with one or two models in production.

1. 1.One implementation of each feature, imported by both the training job and the service. This is free, removes most skew, and is the thing teams skip.
2. 2.Preprocessing inside the saved model artefact, so the service cannot compute it differently even if it wants to.
3. 3.A registry with an alias, for rollback. A day of work using whatever your tracking tool already ships.
4. 4.Monitoring on inputs and predictions. See the drift section for what to alert on and what to merely record.
5. 5.Feature history with validity timestamps, once you have a label that is not scored at the moment the features are computed.
6. 6.A feature store product, when you have several models sharing features and a latency budget that a warehouse query cannot meet. Not before.

What to skip#

• Skip the feature store until two models share a feature. Before that it is a second copy of your warehouse with an on-call rota attached.
• Skip writing the feature twice and testing that the two agree. Testing that two implementations match is more work than having one, and the test decays.
• Worth memorising: preprocessing belongs inside the artefact, unknown-category counts belong on the dashboard, and a feature joined at today's value is a leak.