Asked about Regularisation and validation, hands on
Your model scores 0.95 AUC in cross-validation and 0.70 in production. What might have happened?
An answer that works
Most often leakage in the validation setup: the same entity on both sides of the split, preprocessing fitted on all data, a feature that is only known after the outcome, or shuffling time-ordered data. Otherwise, production data differs from the training data.
Then they push further
How would you check for the same entity on both sides?
Look for an id such as user, patient or device that repeats across rows. Rerun cross-validation with group folds on that id. A large drop means the earlier score was inflated.
And for a feature that leaks the answer?
Check feature importance for anything suspiciously strong, and for each top feature ask when it becomes known. If it is only recorded after the outcome, it cannot be used.
What gets marked up
Strong answers go straight to the validation setup and name group, time and preprocessing leakage. 'The model overfit' without a mechanism is marked down.
Reading an answer is not being able to give one. The section that teaches this is Regularisation and validation, hands on, 5 hours of Overfitting & regularisation.
Learn this properly