Asked about Metrics and thresholds, worked through
Your fraud model has 0.97 ROC AUC. The fraud team says it is useless. Who is right?
An answer that works
Possibly both. With fraud at 0.2%, even a low false positive rate produces far more false alerts than real fraud, so precision at the operating threshold can be very low. I would look at the precision-recall curve and at precision and alert volume at the threshold actually in use.
Then they push further
How would you choose the threshold?
From costs: the average loss from a missed fraud against the cost of reviewing an alert, plus any capacity limit on the team. Pick the threshold that minimises expected cost on validation data, then confirm on held-out data.
The team can only review 500 alerts a day. Does that change anything?
Yes. Then the question becomes which 500 to review, so rank by score and take the top 500. The metric that matters is precision in the top 500, and improving the model means pushing more fraud into that set.
What gets marked up
Strong answers explain why ROC AUC flatters rare-event models and move to precision at the operating point and costs. Treating AUC as the final answer is marked down.
Reading an answer is not being able to give one. The section that teaches this is Metrics and thresholds, worked through, 4 hours of Imbalanced and messy data.
Learn this properly