Model evaluation & metrics interview questions
8 questions across 4 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
Choosing the metric that matches the cost of being wrong
Read the section that teaches thisIf this comes up in an interview
When would you optimise for recall over precision?
When a miss costs more than a false alarm: disease screening, fraud, safety. When a false alarm is expensive, such as a spam filter deleting real mail, precision matters more. The metric follows from the asymmetry of the costs.
Why can ROC AUC mislead on imbalanced data?
It includes the true negative rate, and with overwhelming negatives that stays high regardless, flattering the score. Precision-recall AUC ignores true negatives and is the honest summary when positives are rare.
What is data leakage and how does it happen?
Information from the test set influencing training. Common routes are fitting a scaler or imputer before splitting, features built from future information, duplicate rows across splits, and shuffling a time series. All of them inflate the score and produce a model that fails in production.
How do you choose a classification threshold?
From the business requirement rather than the default of 0.5. Plot the precision-recall curve and pick the point meeting the constraint, such as maximum recall while precision stays above a required level.
Regression metrics and connecting to outcomes
Read the section that teaches thisIf this comes up in an interview
When would you use RMSE rather than MAE?
When large errors are disproportionately costly, since RMSE squares errors before averaging and so penalises big misses heavily. MAE treats all errors equally and is easier to interpret.
Why can MAPE be misleading?
It divides by the actual value, so it becomes huge or undefined when actual values are near zero, and it treats over- and under-prediction asymmetrically.
Metrics in code, with the traps
Read the section that teaches thisIf this comes up in an interview
When would you use macro F1 instead of accuracy?
When classes are imbalanced and each one matters. Accuracy is dominated by the common classes; macro F1 averages per-class F1 equally, so a model that ignores a rare class is penalised.
Then they push further
Your weighted F1 is 0.91 and macro F1 is 0.62. What does that tell you?
Common classes are handled well and at least one rare class badly. Look at per-class scores and the confusion matrix to find which, then consider more data for it, class weights, or whether the class definition is clear.
What gets marked up. Marked up for explaining the gap between averages and moving straight to per-class analysis.
Choosing a metric: four worked cases
Read the section that teaches thisIf this comes up in an interview
You are building a model to flag possibly fraudulent insurance claims for investigation. Which metric do you optimise?
Start from the process: investigators can review a fixed number of claims a week, and a missed fraud costs far more than a review. So the question is how much fraud is caught in the top N flagged claims, precision and recall at that capacity, or better the expected money recovered minus review cost at that operating point.
Then they push further
The business asks for 'accuracy'. What do you say?
That with fraud rare, a model that flags nothing is highly accurate and useless. Then propose the capacity-based metric in their terms: how much fraud the team would catch per week.
What gets marked up. Strong answers start from the downstream action and capacity, not from a metric name.
These are the questions for one topic. The whole question bank covers every topic on the site.