Metrics: which one, and when

The classification and regression metrics, what each one hides, and how to choose between them from the cost of being wrong rather than from habit.

The threshold metrics

Precision	TP ÷ (TP + FP) — of what you flagged, how much was right	The metric when a false positive is expensive: blocking a legitimate transaction, paging an engineer, deleting a file.
Recall (sensitivity)	TP ÷ (TP + FN) — of what was there, how much you found	The metric when a miss is expensive: a missed tumour, a missed fraud, a missed safety violation.
F1	2PR ÷ (P + R) — the harmonic mean	A convenience when precision and recall matter roughly equally. It encodes a specific and often wrong assumption that they do; if you know the relative costs, use them instead.
Accuracy	(TP + TN) ÷ everything	Fine at balanced classes, actively misleading otherwise. At a 1% positive rate, predicting 'no' every time scores 99%.
Specificity	TN ÷ (TN + FP) — of the negatives, how many you left alone	Recall's mirror image. Quoted in medical and screening contexts far more than in machine learning, and it is the same quantity as one minus the false positive rate.
Macro versus micro average	macro weights each class equally; micro weights each example equally	Macro is what you want when rare classes matter. Micro average equals accuracy in single-label multi-class problems, which surprises people.

ROC AUC	probability a random positive scores above a random negative · 0.5 is chance	Threshold-free, so it measures the ranking rather than a decision. Insensitive to class balance, which is a virtue when comparing across datasets and a trap when the positive class is rare.
PR AUC (average precision)	baseline equals the positive rate, not 0.5	Preferred under heavy imbalance. A 0.99 ROC AUC and a 0.35 PR AUC describe the same model at a 1% positive rate, and the second one is the honest number.
Precision at k	of the top k you acted on, how many were right	The right metric whenever capacity is fixed — a review queue, a page of results, an alert budget. Choose k from the real capacity.
Recall at k	of everything relevant, how much appeared in the top k	The metric for the retrieval stage of a RAG system, where a reranker gets a second chance but only over what was retrieved.
NDCG	ranking quality with graded relevance and position discounting	Use when results have degrees of relevance rather than being right or wrong, and when position matters. Overkill when relevance is binary.

Calibration	of the cases predicted at 0.7, about 70% should be positive	Matters whenever a downstream decision uses the number itself — expected value, routing, triage. A model can rank perfectly and be badly calibrated.
Expected calibration error (ECE)	average gap between confidence and accuracy across bins	Bin count changes the value, so quote it with the binning. A reliability diagram tells you more than the scalar does.
Brier score	mean squared error of the predicted probability · lower is better	Proper scoring rule, so it rewards honest probabilities. Decomposes into calibration and discrimination, which is useful when you want to know which one broke.
Log loss	punishes confident wrong answers severely	The loss you usually train against. Unbounded, so one very confident mistake can dominate the mean on a small eval set.

MAE	mean absolute error · in the units of the target	Robust to outliers, and directly interpretable to someone outside the team. Optimising it targets the median.
RMSE	root mean squared error · punishes large misses disproportionately	Use when a single large error is much worse than several small ones. Optimising it targets the mean, which outliers move.
MAPE	mean absolute percentage error	Undefined at zero and asymmetric — it punishes over-prediction more than under-prediction. Common in forecasting and worth replacing with MASE or a scaled error where you can.
Choosing the threshold	where expected cost is lowest, not at 0.5	If a false negative costs ten times a false positive, act whenever predicted probability exceeds roughly 1/11, not 1/2. This single step improves more systems than model changes do.
Choosing the metric	write down what each kind of mistake costs, then pick the metric that reflects it	The question 'which metric' is almost always the question 'what happens when we are wrong, and to whom' in disguise.

Figures last checked 2026-09-13. Several of these age; check anything you are about to rely on.

Where these numbers come from