Asked about Operating the service day to day
What breaks a model in production that no offline evaluation can catch?
An answer that works
Three things, and they need different defences. Feature skew, where serving computes a feature differently from training, fixed by one implementation and preprocessing saved inside the artefact. A stale feature store, where the values are plausible but older than the request, which costs whatever the fast-moving part of the feature was worth. And a dependency upgrade changing preprocessing, which offline evaluation cannot see because the evaluation runs on the new library too, so both sides move together and agree.
Then they push further
How do you defend against the dependency case specifically?
A golden fixture: a handful of representative rows committed to the repository, the preprocessing output hashed including the column names, and the build failing when the hash moves. Regenerate the hash only as a reviewed change, never by rerunning the generator to make a red build pass.
What gets marked up
Marked up for naming all three, and for structural defences rather than more testing of the same kind.
Reading an answer is not being able to give one. The section that teaches this is Operating the service day to day, 4 hours of Deployment & serving.
Learn this properlyAlso asked about Deployment & serving