Choosing a metric: four worked cases

Computing a metric is not the same as knowing which one to optimise, and that choice comes from the problem rather than from a list. This section reasons through four cases where the costs are genuinely different — a spam filter, cancer screening, loan defaults, search results — starting each time from what happens after a prediction, what each kind of error costs, and who pays it. The method transfers to your own problem; the four answers do not.

The method#

Ask what happens after each prediction, what each kind of error costs, and who pays. The metric follows from the answers; it is not chosen from a list.

1  What happens next?      Who or what acts on this prediction, automatically
                           or with a person in the loop?
2  What is the capacity?   How many positives can be acted on per day? A fixed
                           budget turns the question into precision at that k.
3  What does each error    In the units the business already uses: pounds,
   cost, and to whom?      minutes of someone's time, a lost customer.
4  Is the probability      If it is multiplied by anything downstream, you need
   used, or only the       calibration and log loss, not only ranking.
   ordering?

Spam filter        false positive: a real email lost        costly
                   false negative: a spam in the inbox       mildly annoying
                   -> high precision on 'spam'; accept lower recall

Cancer screening   false negative: a missed cancer          very costly
                   false positive: a follow-up test         cheaper
                   -> high recall at an acceptable follow-up rate

Loan default       both errors cost money, in amounts you can estimate
                   -> expected profit at a threshold, from calibrated probabilities

Search results     users look at the top few only
                   -> ranking metrics at small k: precision@5, NDCG@10

The spam filter, in numbers#

The asymmetry here is extreme and worth quantifying, because it explains an operating point that looks absurd from the outside. A spam in the inbox costs someone two seconds. A legitimate invoice silently filed in the junk folder costs a business relationship. Tolerances in real filters are of the order of one legitimate message misfiled in ten thousand, which on a mailbox receiving a hundred good messages a day is one mistake every three years.

That sets the metric: recall on spam, measured at a false positive rate of 0.0001. Not F1, which would happily trade a lost invoice for several caught spams. And it sets what you must measure: at that rate, a test set of a thousand legitimate emails contains an expected 0.1 false positives, so it cannot distinguish a good filter from a terrible one. The evaluation set has to be large enough for the rate you care about, which for one in ten thousand means hundreds of thousands of legitimate messages.

Cancer screening, in numbers#

Run the arithmetic on a screened population of a hundred thousand where the condition is present in five hundred of them. Take a model at ninety percent recall and a ten percent false positive rate, which would be a respectable classifier on most problems.

                        has it (500)     does not (99,500)
flagged                    450                9,950
not flagged                 50               89,550

recall     450 / 500       = 0.90     the number the programme is designed around
precision  450 / 10,400    = 0.043    96 percent of the people recalled are well
accuracy   90,000 / 100,000 = 0.90    identical to the recall, and meaningless here

So the metric is recall at a fixed false positive rate, with the rate set by follow-up capacity rather than by the model. Moving recall from 0.90 to 0.95 sounds small and means fifty more people helped and, at this operating point, several thousand more recalled. Whether that is the right trade is a question for the people who run the programme, and the model's job is to present it as a curve rather than a single number.

Loan defaults, in numbers#

This is the case where there is a right answer and it is arithmetic. Both errors cost money, the amounts are known, so the metric is expected profit and the threshold falls out of the two costs rather than being tuned.

def expected_profit(p_default, gain=1200.0, loss=8000.0):
    """gain: interest earned if the loan is repaid.
    loss: the amount written off if it is not."""
    return (1 - p_default) * gain - p_default * loss

for p in (0.05, 0.10, 0.13, 0.15, 0.30):
    print(p, round(expected_profit(p), 1))

print("break even at p =", round(1200 / (1200 + 8000), 4))

# 0.05   740.0
# 0.1    280.0
# 0.13     4.0
# 0.15  -180.0
# 0.3  -1560.0
# break even at p = 0.1304

The catch is in the word probability. This only works if a predicted 0.13 means thirteen percent of such loans default, so the model must be calibrated and not merely well-ranked. That is why logistic regression and calibrated gradient boosting persist in credit scoring: the number is used arithmetically, so ROC AUC is the wrong thing to report and log loss with a calibration table is the right one.

Search results, in numbers#

Users look at the first few results and abandon rather than scroll, so anything computed over a hundred results is measuring something nobody sees. Report NDCG at 10 as the headline and precision at 3 alongside it, because the first three are the ones that decide whether the user tries again or leaves.

Two guardrails belong next to it. Latency, because a result set that is two percent better and two hundred milliseconds slower is worse in every measurable way. And the share of queries returning nothing useful at all, which an average over all queries hides completely: a system with a good mean NDCG and eight percent of queries returning rubbish feels broken, because the queries that fail are the ones people remember.

What transfers#

• The metric is downstream of the decision. Name the action and the capacity first, and the metric is usually forced.
• One number is never enough. Report the pair that can be traded against each other, plus the guardrail that stops someone gaming it.
• If the probability is used arithmetically, calibration is not optional and ranking metrics are silent about it.
• Make sure the test set is large enough for the operating point. A rate of one in ten thousand cannot be measured on a thousand examples, and people try.