MLOps & model lifecycle interview questions
7 questions across 4 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Keeping a model working after launch
Read the section that teaches thisIf this comes up in an interview
What is model drift and how do you detect it?
Data drift is the inputs changing; concept drift is the relationship between inputs and outputs changing. Detect it by monitoring input and prediction distributions against a fixed reference, since labels often arrive late or never, so accuracy is not available in real time.
Then they push further
Your AUC has not moved in six months. Does that mean nothing has drifted?
No. AUC depends only on the ranking, so it is invariant to any monotone rescaling of the scores and cannot see calibration drift at all. A base rate that moves from 10 per cent to 40 per cent leaves AUC identical to four decimal places while the predicted probabilities become wrong by a quarter, which breaks every expected-value decision downstream. Watch mean predicted probability against realised rate as well as a ranking metric.
Why compare against a fixed reference rather than against last week?
Because a distribution moving one per cent a day never trips a week-on-week comparison and has moved thirty per cent by the end of the month. Hold the reference fixed, usually the validation set the model was signed off on, and move it deliberately when you retrain.
What gets marked up. Marked up for naming both kinds, for monitoring outputs rather than waiting on labels, and for knowing which failures the usual metrics cannot see.
What do you need to version to reproduce a model?
Code, data, hyperparameters and environment. Data is the one people omit and the one that makes reproduction impossible, so datasets need a content hash or an immutable versioned path.
Then they push further
We version the model file itself. Is that not enough?
It lets you serve the old weights again and nothing else. You cannot say why they behaved differently, cannot rebuild them, and cannot produce a like-for-like comparison with the replacement, so the model is effectively frozen: whatever it does now is what it will always do.
How would you deploy a new model version safely?
Shadow deployment first, running it on live traffic without serving its output so you can compare per request rather than by distribution. Then a canary on a small share with guardrails, then a full rollout, with the previous version one alias move away.
Then they push further
What makes a rollback real rather than theoretical?
That it is one action, performed without a deploy, by someone who did not build the system. In practice that means the model is behind a registry alias. The test is not whether a plan exists but whether anyone has actually done it this quarter.
The platform around the model
Read the section that teaches thisIf this comes up in an interview
What is training-serving skew?
Features computed differently at training and serving time, so the deployed model sees inputs unlike its training data. It degrades accuracy with no error, and sharing one implementation of each feature, with the preprocessing saved inside the model artefact, is the fix.
Then they push further
Give an example that an offline evaluation would not catch.
A service that fits its scaler on the incoming batch instead of loading the one fitted during training. On a 20,000-row evaluation batch it is worth three ten-thousandths of AUC, because the batch statistics match the training statistics. Online, requests arrive one at a time, fitting a scaler to one row centres it on itself, and every customer gets the same prediction, wrong by about 0.25. The bug only appears in the regime the evaluation never ran in.
How would you find skew that is already in production?
Log the feature vector as the service computed it, sample it, and recompute the same features through the training path for the same keys and timestamps. Compare distributions per feature. Any feature whose serving and training values differ materially is skewed, and the comparison is worth running as a scheduled job rather than once.
What gets marked up. Marked up for a concrete mechanism, for noticing that offline evaluation cannot see it, and for a structural fix rather than a promise to be careful.
What does a feature store give you?
One computation of each feature served to both training and inference, removing skew by construction, plus point-in-time correct training sets so features reflect only what was known when each example occurred.
Monitoring drift in practice
Read the section that teaches thisIf this comes up in an interview
How would you monitor a fraud model whose labels arrive 30 days late?
Monitor inputs and outputs daily against a reference: feature drift with PSI or similar, the score distribution, the alert rate and data quality. Use proxies such as analyst overturn rates. When labels arrive, compute precision and recall by prediction cohort and compare with the validation baseline.
Then they push further
PSI on one feature jumps to 0.4 overnight. What do you do?
First check for a data problem: an upstream change in units, encoding or a broken join, which is the most common cause. If the data is genuine, check whether predictions shifted and whether the affected segment matters, then decide between waiting for labels, recalibrating or retraining.
What gets marked up. Marked up for checking upstream data problems first, and for cohort-based evaluation once labels arrive.
Serving a model: batch, online, shadow
Read the section that teaches thisIf this comes up in an interview
How would you roll out a retrained model?
Validate offline against the current model on the same held-out data, register it, run it in shadow on live traffic and compare outputs, then canary a small share with business metrics and guardrails, then roll out fully with a fast rollback through the registry.
Then they push further
In shadow, the new model flags 40% more customers. Is that bad?
Not necessarily, but it must be explained before rollout. It may be better calibrated, or it may have learned from a data problem. Look at the extra cases, check calibration on recent labelled data, and consider the operational impact of 40% more actions.
What gets marked up. Marked up for shadow before canary, a registry for rollback, and investigating distribution shifts rather than shipping on offline metrics.
These are the questions for one topic. The whole question bank covers every topic on the site.