Asked about Choosing a metric: four worked cases

You are building a model to flag possibly fraudulent insurance claims for investigation. Which metric do you optimise?

An answer that works

Start from the process: investigators can review a fixed number of claims a week, and a missed fraud costs far more than a review. So the question is how much fraud is caught in the top N flagged claims, precision and recall at that capacity, or better the expected money recovered minus review cost at that operating point.

Then they push further

The business asks for 'accuracy'. What do you say?

That with fraud rare, a model that flags nothing is highly accurate and useless. Then propose the capacity-based metric in their terms: how much fraud the team would catch per week.

What gets marked up

Strong answers start from the downstream action and capacity, not from a metric name.

Reading an answer is not being able to give one. The section that teaches this is Choosing a metric: four worked cases, 3 hours of Model evaluation & metrics.

Learn this properly

Also asked about Model evaluation & metrics

When would you optimise for recall over precision?Why can ROC AUC mislead on imbalanced data?What is data leakage and how does it happen?How do you choose a classification threshold?When would you use RMSE rather than MAE?Why can MAPE be misleading?When would you use macro F1 instead of accuracy?