Asked about Serving a model: batch, online, shadow
How would you roll out a retrained model?
An answer that works
Validate offline against the current model on the same held-out data, register it, run it in shadow on live traffic and compare outputs, then canary a small share with business metrics and guardrails, then roll out fully with a fast rollback through the registry.
Then they push further
In shadow, the new model flags 40% more customers. Is that bad?
Not necessarily, but it must be explained before rollout. It may be better calibrated, or it may have learned from a data problem. Look at the extra cases, check calibration on recent labelled data, and consider the operational impact of 40% more actions.
What gets marked up
Marked up for shadow before canary, a registry for rollback, and investigating distribution shifts rather than shipping on offline metrics.
Reading an answer is not being able to give one. The section that teaches this is Serving a model: batch, online, shadow, 4 hours of MLOps & model lifecycle.
Learn this properlyAlso asked about MLOps & model lifecycle