Keeping a model working after launch

Shipping a model is not the end of the work, because the world it was trained on keeps moving. This section is about what happens afterwards: knowing which version is live and what data produced it, watching for the two ways incoming data can drift away from what the model expects, and deciding in advance what triggers a retrain, so the decision is not made in an incident.

Deploying a model and owning one#

Deploying a model is a finite task: package it, put it behind an endpoint, check the endpoint answers. It ends. Owning one does not end, and the difference is the whole of this topic. An owner can answer five questions at any hour of any day, and the machinery in this section exists so that the answers come from a record rather than from someone's memory.

1. 1.Which model is serving right now, and which run produced it? A version string returned with every prediction, resolvable to a run.
2. 2.What data was it trained on, and can that data still be fetched? An immutable snapshot with a content hash, not a table name.
3. 3.How would I know it had stopped working? A monitor on inputs and outputs that does not wait for labels.
4. 4.What would make me retrain it, decided before anyone is under pressure? A written trigger with a number in it.
5. 5.How fast can I put the previous version back, and has anyone done it this quarter? A registry alias, and a drill.

A team that cannot answer all five has deployed a model and has not taken ownership of one. The gap shows up months later, in the form of a business number moving the wrong way and nobody able to say what changed.

A model is not a build artefact#

Ordinary code behaves the same on day 400 as on day 1. A model degrades, because the world it was trained on moves away from the world it now sees. Nothing errors when this happens, which is what makes it dangerous. A web service that breaks returns a 500 and pages someone; a model that breaks returns a confident number that is wrong, at the same latency as before, with a 200 status code.

Version the model, the data and the code together#

To reproduce a model you need the code, the data, the hyperparameters and the environment. Missing any one of them and a result from three months ago is unrecoverable. The common half-measure is to version only the model file, which buys nothing you actually want: you can serve the old weights again, but you cannot say why they behaved differently, cannot retrain them, and cannot produce a like-for-like comparison with the replacement.

model_v14
  code       git sha a3f21e
  data       s3://datasets/training/2026-03-01/ (hash 9c2b...)
  config     lr 3e-4, batch 64, 12 epochs
  env        image ml-base:2026.03
  metrics    val 0.874, test 0.861

The three parts fail in different ways and it is worth knowing which failure you are buying. A versioned model with no code version means you can serve it and never change it, because nobody can rebuild the preprocessing it expects. A versioned model with no data version means you can rebuild the pipeline and get a different model, because the training table has been overwritten twice since. A versioned model with no environment means it loads on someone's laptop and not on the training box, usually because a serialisation format changed under you. Version all three or accept that the model is frozen: whatever it does now is what it will always do.

The practical form of this is a registry entry, not a document. A registry stores each trained model with that metadata attached and a mutable alias pointing at the one that is live, so promoting a model and rolling one back are both recorded actions rather than someone copying a file onto a server. The platform section covers what that looks like in code.

Two kinds of drift#

• Data drift: the inputs change. New product categories, a different user population, an upstream format change. The relationship may still hold.
• Concept drift: the relationship changes. What predicted fraud last year does not now, because the fraud adapted. More serious and harder to detect.

What to monitor#

Input distribution      has the shape of incoming data moved
Prediction distribution has the balance of outputs shifted
Confidence              are predictions getting less certain
Latency and errors      ordinary service health
Accuracy                where labels eventually arrive
Business metric         the thing the model was meant to improve

The prediction distribution is the cheapest useful signal. If a fraud model flagged 2 percent of transactions for six months and now flags 8, something changed, and you know it today rather than when the labels arrive.

Retraining: schedule or trigger#

There are two policies and the argument between them is usually conducted without numbers. On a schedule is simple and predictable and retrains when nothing changed. On a trigger is efficient and needs the monitoring to work. The way to settle it is to simulate the drift you actually expect and price both. Here is that simulation: a binary classifier launched on month zero, with the true relationship rotating over eighteen months so that the weight on one feature decays away and a feature the model has never used becomes important. Concept drift, in other words, not data drift — the inputs look identical throughout.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(7)
W0 = np.array([1.4, -0.9, 0.6, 0.0, 0.0])          # the true coefficients at launch

def month(t, n=6000, rng=rng):
    """Concept drift: the weight on feature 3 rotates into feature 4 over 18 months."""
    w = W0.copy()
    w[3] = 1.2 * min(t / 18.0, 1.0)
    w[2] = 0.6 * (1 - min(t / 18.0, 1.0))
    X = rng.normal(0, 1, size=(n, 5))
    p = 1 / (1 + np.exp(-(X @ w)))
    return X, (rng.random(n) < p).astype(int)

Xtr, ytr = month(0, 20000)
frozen = LogisticRegression(max_iter=1000).fit(Xtr, ytr)
rolling = LogisticRegression(max_iter=1000).fit(Xtr, ytr)
trigger = LogisticRegression(max_iter=1000).fit(Xtr, ytr)
retrains_r = retrains_t = 0

print("month  frozen  monthly  trigger   retrained")
for t in range(1, 19):
    X, y = month(t)
    a_f = roc_auc_score(y, frozen.predict_proba(X)[:, 1])
    a_r = roc_auc_score(y, rolling.predict_proba(X)[:, 1])
    a_t = roc_auc_score(y, trigger.predict_proba(X)[:, 1])
    fired = a_t < 0.80                                # the trigger: AUC below the floor
    print(f"{t:>5} {a_f:>7.4f} {a_r:>8.4f} {a_t:>8.4f}   {'retrain' if fired else '':>7}")
    rolling = LogisticRegression(max_iter=1000).fit(X, y)
    retrains_r += 1
    if fired:
        trigger = LogisticRegression(max_iter=1000).fit(X, y)
        retrains_t += 1
print(f"retrains: monthly {retrains_r}, trigger {retrains_t}")

month  frozen  monthly  trigger   retrained
    1  0.8406   0.8406   0.8406
    4  0.8346   0.8406   0.8346
    8  0.8187   0.8359   0.8187
   11  0.8012   0.8410   0.8012
   14  0.7977   0.8563   0.7977   retrain
   15  0.7960   0.8654   0.8654
   18  0.7700   0.8649   0.8612
retrains: monthly 18, trigger 1

Three things to take from that table. The frozen model lost 7 points of AUC over eighteen months, 0.8406 to 0.7700, at no point failing in any way a service monitor would see. Monthly retraining held it between 0.83 and 0.87 throughout, at a cost of eighteen retrains. The trigger arm retrained once, in month fourteen, and finished at 0.8612 against monthly retraining's 0.8649 — essentially the same place for one-eighteenth of the training compute.

The cost of the trigger is the thirteen months it spent below the monthly arm before it fired, bottoming out at 0.7977. Whether that matters is a business question, not a modelling one, and it is the question to ask before choosing. The fix, when it does matter, is not to abandon triggers but to set the threshold relative to launch rather than at a round number: a trigger at three points below the launch AUC of 0.8406, so 0.8106, would have fired at month eleven instead of fourteen. An absolute floor of 0.80 is a threshold nobody derived, and it is the most common thing on these dashboards.

• Schedule when retraining is cheap, labels are plentiful and the world moves steadily. It needs no monitoring to work and it cannot be forgotten.
• Trigger when retraining is expensive, needs human review, or the data has a seasonal shape that a schedule would chase.
• Either way, validate the candidate against the incumbent on the same held-out window before promoting. A retrained model is not automatically better, and a retrain during an unusual period can be substantially worse.
• Log the trigger's inputs whether or not it fires, so you can see afterwards how close it came.

Rolling out, and rolling back#

Shadow deployment runs the new model alongside the old on real traffic without serving its output, so you can compare on live data before risking anything. Then a canary on a small share of traffic, then a full rollout. The serving section works through both in code.

Rollback is the half that gets designed last and used first. It has to be one action that a person who did not build the system can perform at three in the morning, which in practice means moving a registry alias back and restarting nothing. Two things break it. The first is state written by the new model that the old one cannot read — a new feature column, a changed score scale that downstream thresholds were retuned against — so rolling the model back leaves the system in a configuration neither version was tested in. The second is never having done it. A rollback path that has not been exercised in a drill is a plan, not a capability.

What to skip#

• Do not build automated retraining before you have monitoring. Automatic retraining on drifted data with no validation gate is a machine for deploying worse models quickly.
• Do not monitor every feature. The drift section shows what happens to a dashboard with fifty statistics on it.
• Do not version the model file alone. It is the cheapest of the three things to store and the least useful on its own.
• Worth memorising: the five ownership questions, the two kinds of drift, and that a schedule needs no monitoring to work while a trigger needs it to work perfectly.