Asked about Monitoring drift in practice
How would you monitor a fraud model whose labels arrive 30 days late?
An answer that works
Monitor inputs and outputs daily against a reference: feature drift with PSI or similar, the score distribution, the alert rate and data quality. Use proxies such as analyst overturn rates. When labels arrive, compute precision and recall by prediction cohort and compare with the validation baseline.
Then they push further
PSI on one feature jumps to 0.4 overnight. What do you do?
First check for a data problem: an upstream change in units, encoding or a broken join, which is the most common cause. If the data is genuine, check whether predictions shifted and whether the affected segment matters, then decide between waiting for labels, recalibrating or retraining.
What gets marked up
Marked up for checking upstream data problems first, and for cohort-based evaluation once labels arrive.
Reading an answer is not being able to give one. The section that teaches this is Monitoring drift in practice, 4 hours of MLOps & model lifecycle.
Learn this properly