Imbalanced and messy data interview questions
5 questions across 2 sections, in the order you would learn them. Every answer here is one that works — but an answer you have read is not an answer you can give, so use Test yourself and say it out loud first.
When one class is rare
Read the section that teaches thisIf this comes up in an interview
Your classes are 99 to 1. What do you do?
Stop using accuracy, and use precision, recall and PR AUC. Try class weights first, tune the decision threshold from the precision-recall curve rather than leaving it at 0.5, and evaluate on the real distribution rather than a rebalanced one.
Should you oversample before or after splitting?
After, and only the training fold. Oversampling before splitting puts copies of the same example in both train and test, which inflates the score. The validation set must keep the real class balance.
How do you pick the decision threshold?
From the cost of each error type. Compute expected cost across thresholds using the real cost of a miss and of a false positive, and pick the minimum. That turns a metric argument into an arithmetic answer.
Metrics and thresholds, worked through
Read the section that teaches thisIf this comes up in an interview
Your fraud model has 0.97 ROC AUC. The fraud team says it is useless. Who is right?
Possibly both. With fraud at 0.2%, even a low false positive rate produces far more false alerts than real fraud, so precision at the operating threshold can be very low. I would look at the precision-recall curve and at precision and alert volume at the threshold actually in use.
Then they push further
How would you choose the threshold?
From costs: the average loss from a missed fraud against the cost of reviewing an alert, plus any capacity limit on the team. Pick the threshold that minimises expected cost on validation data, then confirm on held-out data.
The team can only review 500 alerts a day. Does that change anything?
Yes. Then the question becomes which 500 to review, so rank by score and take the top 500. The metric that matters is precision in the top 500, and improving the model means pushing more fraud into that set.
What gets marked up. Strong answers explain why ROC AUC flatters rare-event models and move to precision at the operating point and costs. Treating AUC as the final answer is marked down.
What does it mean for a classifier to be calibrated, and when do you need it?
Its scores match observed frequencies: cases scored 0.3 are positive about 30% of the time. You need it whenever the score is used as a probability, such as in expected cost calculations or when showing risk to people.
These are the questions for one topic. The whole question bank covers every topic on the site.