Asked about The platform around the model
What is training-serving skew?
An answer that works
Features computed differently at training and serving time, so the deployed model sees inputs unlike its training data. It degrades accuracy with no error, and sharing one implementation of each feature, with the preprocessing saved inside the model artefact, is the fix.
Then they push further
Give an example that an offline evaluation would not catch.
A service that fits its scaler on the incoming batch instead of loading the one fitted during training. On a 20,000-row evaluation batch it is worth three ten-thousandths of AUC, because the batch statistics match the training statistics. Online, requests arrive one at a time, fitting a scaler to one row centres it on itself, and every customer gets the same prediction, wrong by about 0.25. The bug only appears in the regime the evaluation never ran in.
How would you find skew that is already in production?
Log the feature vector as the service computed it, sample it, and recompute the same features through the training path for the same keys and timestamps. Compare distributions per feature. Any feature whose serving and training values differ materially is skewed, and the comparison is worth running as a scheduled job rather than once.
What gets marked up
Marked up for a concrete mechanism, for noticing that offline evaluation cannot see it, and for a structural fix rather than a promise to be careful.
Reading an answer is not being able to give one. The section that teaches this is The platform around the model, 4 hours of MLOps & model lifecycle.
Learn this properlyAlso asked about MLOps & model lifecycle