Forecasting in code: baselines, backtests and intervals
By the end: You can backtest forecasts properly, compare against a seasonal naive baseline with a scale-free metric, and produce prediction intervals.
- The seasonal naive baseline
- Rolling-origin backtesting
- MASE: a metric that compares across series
- Lag features without leakage
- Quantile models for intervals
New to time series? Time series data is measurements taken in order over time: daily sales, server latency every minute, a patient's readings across a year. The order is the information. Each value is related to the ones just before it, there are cycles that repeat by weekday and by season, and there is usually a slow trend underneath. Forecasting is predicting what the next values will be. Why it matters
Forecasting is where a careless evaluation fools you most easily, because the ordinary way of splitting data lets the model see the future. This section sets up the discipline instead: the seasonal naive baseline you have to beat before anything clever is worth trying, backtesting that respects the order of time, a metric that can be compared across series of different sizes, lag features built without leakage, and quantile models for when a range matters more than a single number.
Beat this first#
def seasonal_naive(history, horizon, season=7):
"""Forecast each future day as the same weekday last week."""
last_season = history[-season:]
return [last_season[i % season] for i in range(horizon)]For daily data with a weekly pattern, this is surprisingly hard to beat. Any model that does not beat it clearly is not worth its complexity.
Clearly is doing work in that sentence. A model that beats seasonal naive by three percent on one test window has not beaten it, because the fold-to-fold spread is usually larger than three percent. Run several origins, look at the spread, and treat a win smaller than the spread as no result. This is also the honest way to end a project early: if nothing beats the baseline after a fortnight, the baseline is the deliverable, and a seasonal naive forecast in production costs nothing to run and never has an outage.
Which baseline to use depends on the series, and picking the flattering one is a way of cheating that is easy to do by accident. Use seasonal naive on anything with a weekly or yearly shape. Use plain naive, tomorrow equals today, on anything that looks like a random walk, which includes most prices and most financial series and which seasonal naive loses to badly. If you are unsure, compute all of them; they are three lines each and the most expensive one to skip is the one you would have lost to.
Backtest like you will forecast#
train ------------------|test 14 days|
train ----------------------------|test 14 days|
train --------------------------------------|test 14 days|
Each fold trains only on the past, forecasts the next 14 days, then the origin moves forward.One split tells you how the model did in one fortnight, which may have been unusual. Several rolling origins show how it does across conditions, and match how the model will be used: retrained and forecasting forward repeatedly.
import numpy as np
def rolling_origin(series, forecaster, score, horizon=14, folds=5,
step=14, min_train=120):
"""Each fold trains on everything up to its origin and scores the next
horizon days. The origin moves forward by step each time."""
out = []
for f in range(folds):
end = min_train + f * step
if end + horizon > len(series):
break
history, actual = series[:end], series[end : end + horizon]
out.append((end, score(actual, forecaster(history, horizon), history)))
return outA metric that means something#
import numpy as np
def mase(actual, forecast, history, season=7):
"""Error relative to the in-sample seasonal naive error. Below 1 beats the baseline."""
naive_err = np.mean(np.abs(history[season:] - history[:-season]))
return np.mean(np.abs(actual - forecast)) / naive_errPercentage errors explode when actual values are near zero, and plain error cannot be compared across a high-volume and a low-volume product. Scaling by the naive error fixes both, and says directly whether you beat the baseline.
Put the two together and the value of several origins becomes visible immediately. The same forecaster, on the same series, scores between 0.74 and 1.39 depending only on which fortnight it was asked about.
import numpy as np
def seasonal_naive(history, horizon, season=7):
last = history[-season:]
return np.array([last[i % season] for i in range(horizon)])
def flat(history, horizon):
return np.full(horizon, history[-1])
rng = np.random.default_rng(0)
t = np.arange(260)
series = 100 + 0.1 * t + 12 * np.sin(2 * np.pi * t / 7) + rng.normal(0, 4, 260)
for end, m in rolling_origin(series, seasonal_naive, mase):
print("train to", end, "MASE", round(m, 3))
scores = [m for _, m in rolling_origin(series, seasonal_naive, mase)]
flat_scores = [m for _, m in rolling_origin(series, flat, mase)]
print("seasonal naive mean", round(float(np.mean(scores)), 3))
print("flat naive mean ", round(float(np.mean(flat_scores)), 3))
# train to 120 MASE 0.828
# train to 134 MASE 1.197
# train to 148 MASE 0.74
# train to 162 MASE 1.388
# train to 176 MASE 0.798
# seasonal naive mean 0.99
# flat naive mean 1.854Lags without leakage#
When forecasting 14 days ahead, a feature like 'yesterday's sales' is not available for day 14 at forecast time. Either build features only from data at least as old as the horizon, or forecast one step at a time and feed predictions back in, knowing errors compound.
import numpy as np
def features_for(series, i, horizon):
"""Features for predicting day i, using only data available h days earlier.
Every lag is at least horizon old, so this works for the direct approach."""
cut = i - horizon # the last day you will actually know
if cut < 28:
return None
past = series[:cut + 1]
return {
"lag_h": past[-1], # h days before the target
"lag_h_7": past[-8], # and a week before that
"roll_7": past[-7:].mean(),
"roll_28": past[-28:].mean(),
"vol_28": past[-28:].std(),
"dow": i % 7, # known for any future date
}
series = np.arange(100).astype(float)
print(features_for(series, i=60, horizon=14)["lag_h"]) # 46.0, not 59.0Intervals from quantile models#
from sklearn.ensemble import HistGradientBoostingRegressor
models = {q: HistGradientBoostingRegressor(loss="quantile", quantile=q).fit(X_train, y_train)
for q in (0.1, 0.5, 0.9)}
low, mid, high = (models[q].predict(X_test) for q in (0.1, 0.5, 0.9))
coverage = np.mean((y_test >= low) & (y_test <= high)) # should be near 0.8An inventory planner needs to know how bad it could plausibly be, not just the middle guess. Check that an 80% interval really contains about 80% of actual values in the backtest.
Three quantile models are three separate fits, and nothing forces them to agree. On a hard row the 0.1 model can predict above the 0.9 model, which is called quantile crossing and produces an interval with a negative width. Nobody notices until a downstream system divides by it. Sort the three predictions row by row before using them; it is one line and it is always correct, because a set of quantiles that are out of order was wrong in a way sorting cannot make worse.
import numpy as np
low = np.array([12.0, 30.0, 41.0])
mid = np.array([18.0, 28.0, 45.0]) # the middle model disagrees on row 2
high = np.array([25.0, 27.0, 52.0]) # and the upper one is lowest there
stacked = np.sort(np.column_stack([low, mid, high]), axis=1)
low, mid, high = stacked[:, 0], stacked[:, 1], stacked[:, 2]
print(low, mid, high)
# [12. 27. 41.] [18. 28. 45.] [25. 30. 52.]
print(bool((high >= low).all())) # True, now guaranteedThen report coverage per horizon, not pooled. Intervals are usually about right one day ahead and far too narrow at thirty, because the model's uncertainty about its own level grows with the horizon and a fixed-width interval does not. A single pooled coverage of 79 percent can hide 95 percent at day one and 55 percent at day thirty, and day thirty is the one the planner is sizing stock against.
Where people go wrong
Using lag-1 features for a 14-day-ahead forecast. The backtest looks superb because it uses information that will not exist at forecast time.
If this comes up in an interview
How would you evaluate a demand forecasting model?
Rolling-origin backtesting that trains only on the past and forecasts the real horizon, a seasonal naive baseline, a scale-free metric such as MASE so products of different volume compare, and interval coverage if the business uses ranges.
Then they push further
Your model beats the baseline on average but loses badly on promotions. What next?
Promotions are a known future event, so add them as features, including planned ones, and evaluate that slice separately. If data is thin, a separate adjustment for promotion periods may beat one global model.
Why not use mean absolute percentage error?
It is undefined or huge when actual values are near zero and penalises over- and under-forecasts asymmetrically. Scaled errors avoid both problems.
What gets marked up. Marked up for rolling backtests, a naive baseline, and a scale-free metric. Random train-test splits are marked down immediately.
Build this: On a public daily sales or traffic series, implement seasonal naive, a gradient boosting model with horizon-safe lags, and quantile intervals. Backtest with five rolling origins and report MASE and interval coverage.
Watch it explained
This is one section of Time series, 17 hours across 4 sections. Read it in full to get the whole topic in order.