Regression metrics and connecting to outcomes
By the end: You can evaluate regression models and translate any metric into a business effect.
- MAE, RMSE and MAPE
- Which errors your metric emphasises
- Offline metrics versus outcomes
- Choosing a primary metric
New to model evaluation & metrics? Once a model makes predictions you need a number saying how good they are. Evaluation is choosing that number and measuring it honestly. Accuracy — the share of predictions that were right — is the obvious choice and often the wrong one: a test for a disease that one person in a thousand has is 99.9% accurate if it simply says no to everybody. Why it matters
Classification metrics get most of the attention, but plenty of what gets predicted is a number: a demand, a duration, a price. This section covers the metrics for that and, more usefully, what each one chooses to punish — RMSE cares about large errors, MAE treats them all alike, MAPE falls apart near zero. The second half is what decides whether your work gets used: translating any of these into an effect somebody outside the team cares about, because the model with the better score can still be the worse one if its errors land on the days that matter.
Regression metrics#
- Mean absolute error: average size of the error. Easy to interpret, treats all errors equally.
- Root mean squared error: penalises large errors heavily. Use it when a big miss is much worse than several small ones.
- Mean absolute percentage error: relative error. Intuitive, and it breaks when actual values are near zero.
Errors: 1, 1, 1, 1, 10
MAE = 2.8 the typical error
RMSE = 4.6 pulled up by the single large missThat gap is a diagnostic worth using deliberately. RMSE is always at least MAE, and the ratio between them tells you the shape of your error distribution: close to one means the errors are all much the same size, and a ratio of two or more means a handful of large misses are carrying the number. Which you should care about depends entirely on whether one big miss costs more than five small ones, and that is a question about the business, not about the model.
import numpy as np
def mae(y, p):
return float(np.mean(np.abs(y - p)))
def rmse(y, p):
return float(np.sqrt(np.mean((y - p) ** 2)))
def mape(y, p):
return float(np.mean(np.abs((y - p) / y))) # undefined at y = 0
def wape(y, p):
"""Weighted absolute percentage error: total error over total actual.
Survives zeros, and is what most demand planners actually want."""
return float(np.abs(y - p).sum() / np.abs(y).sum())
y = np.array([100.0, 100, 100, 100, 100])
spread = np.array([98.0, 102, 99, 101, 100]) # five small misses
lumpy = np.array([100.0, 100, 100, 100, 90]) # one large miss
for name, p in [("spread", spread), ("lumpy", lumpy)]:
print(name, "MAE", round(mae(y, p), 2), "RMSE", round(rmse(y, p), 2),
"ratio", round(rmse(y, p) / mae(y, p), 2))
# spread MAE 1.2 RMSE 1.41 ratio 1.18 errors all alike
# lumpy MAE 2.0 RMSE 4.47 ratio 2.24 one miss is doing the workMAPE deserves its own warning because it is the one most often asked for by name. It divides by the actual, so it is undefined at zero and enormous near it, which is fatal for anything intermittent: a product that sells nothing on Tuesday cannot have a percentage error on Tuesday. Worse, it is asymmetric. Over-predicting by fifty when the actual is a hundred costs 0.5; under-predicting to zero also costs 1.0 at most, whatever the over-prediction, because the error is capped by the actual on one side and unbounded on the other. A model tuned on MAPE learns to forecast low.
import numpy as np
actual = 100.0
for pred in (50.0, 150.0, 0.0, 200.0):
print(pred, round(abs(actual - pred) / actual, 2))
# 50.0 0.5 under by half
# 150.0 0.5 over by half, same penalty, so far so fair
# 0.0 1.0 predict nothing: the worst possible under-prediction costs 1.0
# 200.0 1.0 predict double: the same 1.0, and there is no ceiling above it
print(round(np.mean([abs(0 - 5) / 5, abs(10 - 5) / 5]), 2)) # 1.0
print(round(np.mean([abs(0.1 - 0.0001) / 0.0001, 0]), 1)) # 499.5 from one tiny actualWhen the two directions genuinely cost different amounts, stop trying to find a symmetric metric that captures it and use a loss that does not pretend. The pinball, or quantile, loss weights over- and under-prediction differently by construction, and minimising it produces a forecast of that quantile rather than the mean.
import numpy as np
def pinball(y, p, q):
"""q is the quantile. q=0.9 punishes under-prediction nine times as hard."""
error = y - p
return float(np.mean(np.maximum(q * error, (q - 1) * error)))
y = np.array([100.0] * 5)
print(round(pinball(y, np.full(5, 90.0), 0.9), 2)) # 9.0 under by 10
print(round(pinball(y, np.full(5, 110.0), 0.9), 2)) # 1.0 over by 10, far cheaper
print(round(pinball(y, np.full(5, 90.0), 0.5), 2)) # 5.0 at q=0.5 this is MAE/2Connect to what matters#
A metric is only useful if it moves with something the business cares about. A forecasting model with lower RMSE may still perform worse in practice if its errors fall on the days that matter most, such as the busiest period.
The way to settle that is to compute the thing itself rather than a proxy for it. Attach a cost to each direction of error, sum it over the test period, and compare models on pounds. It takes an afternoon, it usually reorders the models, and it turns an argument about metrics into an arithmetic problem.
import numpy as np
def stock_cost(actual, forecast, holding=1.0, stockout=6.0):
"""Over-forecast: units sit in a warehouse. Under-forecast: a lost sale."""
over = np.maximum(forecast - actual, 0).sum() * holding
under = np.maximum(actual - forecast, 0).sum() * stockout
return float(over + under)
rng = np.random.default_rng(0)
actual = rng.poisson(40, 365).astype(float)
cautious = np.full(365, 40.0) # the mean: the RMSE-optimal answer
padded = np.full(365, 48.0) # deliberately over-forecast
for name, f in [("mean forecast", cautious), ("padded forecast", padded)]:
print(f"{name:16}",
"RMSE", round(float(np.sqrt(np.mean((actual - f) ** 2))), 2),
"| cost", round(stock_cost(actual, f)))
# mean forecast RMSE 6.51 | cost 7310
# padded forecast RMSE 9.77 | cost 3762Translate the metric into money
Being able to say that a 5 percent error reduction avoids a known amount of overstock or a number of missed deliveries is what makes a model's value legible to people who fund it. Get the two unit costs from whoever owns the budget rather than inventing them, write them down in the report, and the conversation about whether the model is worth keeping becomes a conversation about whether those two numbers are right.
Where people go wrong
Choosing a metric because it is standard. The metric should encode which errors cost the most in your application.
If this comes up in an interview
When would you use RMSE rather than MAE?
When large errors are disproportionately costly, since RMSE squares errors before averaging and so penalises big misses heavily. MAE treats all errors equally and is easier to interpret.
Why can MAPE be misleading?
It divides by the actual value, so it becomes huge or undefined when actual values are near zero, and it treats over- and under-prediction asymmetrically.
Build this: Evaluate one regression model with MAE, RMSE and MAPE, then translate the best improvement into an estimated business effect.
Watch it explained
This is one section of Model evaluation & metrics, 16 hours across 4 sections. Read it in full to get the whole topic in order.