Asked about Choosing the metric that matches the cost of being wrong
What is data leakage and how does it happen?
An answer that works
Information from the test set influencing training. Common routes are fitting a scaler or imputer before splitting, features built from future information, duplicate rows across splits, and shuffling a time series. All of them inflate the score and produce a model that fails in production.
Reading an answer is not being able to give one. The section that teaches this is Choosing the metric that matches the cost of being wrong, 5 hours of Model evaluation & metrics.
Learn this properlyAlso asked about Model evaluation & metrics
When would you optimise for recall over precision?Why can ROC AUC mislead on imbalanced data?How do you choose a classification threshold?When would you use RMSE rather than MAE?Why can MAPE be misleading?When would you use macro F1 instead of accuracy?You are building a model to flag possibly fraudulent insurance claims for investigation. Which metric do you optimise?