Data where order carries meaning
By the end: You can forecast without fooling yourself through leakage.
- Trend, seasonality and autocorrelation
- Validation that respects time
- Feature engineering for sequences
- Baselines that are hard to beat
New to time series? Time series data is measurements taken in order over time: daily sales, server latency every minute, a patient's readings across a year. The order is the information. Each value is related to the ones just before it, there are cycles that repeat by weekday and by season, and there is usually a slow trend underneath. Forecasting is predicting what the next values will be. Why it matters
In most machine learning you can shuffle the rows. In time series you cannot: order is the signal, and it is also what makes ordinary validation lie to you. Split a series at random and information from the future leaks into training, so the model scores beautifully in testing and falls apart the first time it forecasts something real. This section covers the structure worth modelling — trend, seasonality, autocorrelation — validation that respects time, and the naive baseline that a surprising number of forecasting projects never beat and never check against.
The structure#
- Trend: the long-run direction.
- Seasonality: repeating patterns at fixed periods, daily, weekly, yearly.
- Autocorrelation: today resembles yesterday. This is what makes forecasting possible and also what makes naive validation wrong.
Autocorrelation is the one to sit with, because both halves of that sentence follow from the same fact. If today resembles yesterday, then yesterday is a useful feature, which is why forecasting works at all. And if today resembles yesterday, then a validation row whose neighbours are in training is not a test of anything: the model can answer it by remembering the days on either side, which is a skill it will never be asked for again.
The mistake that invalidates everything#
Never shuffle a time series
Ordinary cross-validation shuffles rows, which puts future data in the training set and past data in validation. Because consecutive points are similar, the model effectively sees the answer. The score is excellent and completely meaningless. This is the most common serious error in applied machine learning.
The size of that effect is worth measuring rather than believing. Below is a series whose level drifts — a random walk, which is what most business series look like once seasonality is removed. The model is deliberately trivial: predict a day using the nearest day it saw in training. That is roughly what any flexible model does on a series with no other features, so it is a fair stand-in.
import numpy as np
rng = np.random.default_rng(0)
n = 500
level = np.cumsum(rng.normal(0, 1, n)) # a drifting level
y = level + rng.normal(0, 0.5, n)
def nearest_day(train_idx, test_idx):
"""Predict each test day with the value of the closest day in training."""
out = np.empty(len(test_idx))
for j, i in enumerate(test_idx):
k = train_idx[np.argmin(np.abs(train_idx - i))]
out[j] = y[k]
return out
def rmse(a, b):
return float(np.sqrt(np.mean((a - b) ** 2)))
perm = rng.permutation(n)
rand_test, rand_train = np.sort(perm[:100]), np.sort(perm[100:])
print("random split", round(rmse(y[rand_test], nearest_day(rand_train, rand_test)), 2))
time_train, time_test = np.arange(400), np.arange(400, 500)
print("time split ", round(rmse(y[time_test], nearest_day(time_train, time_test)), 2))
# random split 1.33
# time split 5.47from sklearn.model_selection import TimeSeriesSplit, cross_val_score
# Wrong
cross_val_score(model, X, y, cv=5)
# Right: each fold trains on the past and validates on the future
cross_val_score(model, X, y, cv=TimeSeriesSplit(n_splits=5))fold 1 train [-------] test [--]
fold 2 train [----------] test [--]
fold 3 train [-------------] test [--]
fold 4 train [----------------] test [--]
Each fold trains only on what came before its test window, and the training
set grows. This is forward chaining, and it is what TimeSeriesSplit does.Leave a gap when the label arrives late
If the target for day D is only known on day D plus 30 — a churn label, a default, a delivery outcome — then a fold that trains up to the day before its test window trains on labels that would not have existed. Drop thirty days between the end of training and the start of testing. This is called a purged or embargoed split, and forgetting it is how a backtest stays optimistic even after someone has correctly stopped shuffling.
Features#
df["lag_1"] = df.value.shift(1)
df["lag_7"] = df.value.shift(7)
df["rolling_7"] = df.value.shift(1).rolling(7).mean() # shift FIRST
df["dow"] = df.date.dt.dayofweek
df["is_holiday"] = df.date.isin(holidays)That caption is easy to nod at and easy to get wrong at half past six on a Friday, so here are the two columns side by side. The series has one spike, and on the spike day the leaky feature has already noticed.
import numpy as np
values = np.array([10.0, 12, 11, 40, 13, 12, 11])
def rolling_including_today(v, w=3):
return [round(float(v[i - w + 1 : i + 1].mean()), 1) if i >= w - 1 else None
for i in range(len(v))]
def rolling_shifted(v, w=3):
return [round(float(v[i - w : i].mean()), 1) if i >= w else None
for i in range(len(v))]
print("value ", [float(x) for x in values])
print("leaky ", rolling_including_today(values))
print("shifted ", rolling_shifted(values))
# value [10.0, 12.0, 11.0, 40.0, 13.0, 12.0, 11.0]
# leaky [None, None, 11.0, 21.0, 21.3, 21.7, 12.0]
# shifted [None, None, None, 11.0, 21.0, 21.3, 21.7]Which lags to build is a question with a cheap empirical answer: compute the correlation between the series and itself shifted by k, for k from one to about three seasons, and take the peaks. Daily retail data will show a spike at 7 and a smaller one at 14 and 21. Hourly traffic shows 24 and 168. Build lags at those, plus rolling means at a few windows to give the model a smoothed level as well as individual days.
- Lags: the value k steps ago. Cheap, and the single most useful family of features on almost any series.
- Rolling statistics: mean, standard deviation, min and max over a window ending before today. The standard deviation is the one people forget, and it is what lets a model know the series has become volatile.
- Differences: value minus the value a season ago. This removes the level, which matters because a tree model cannot extrapolate beyond the range of levels it was trained on.
- Calendar: day of week, day of month, month, holiday, days to the next holiday, and any known future event such as a planned promotion. These are the only features you are certain to have at forecast time.
Seasonality#
Seasonality is the part of the series that repeats on a known period, and the useful thing about it is that it is knowable in advance. Separate it from the trend before you model anything, because what looks like a model failing on Saturdays is usually a weekly pattern nobody gave it.
import numpy as np
rng = np.random.default_rng(2)
n = 364
t = np.arange(n)
dow = t % 7
weekly = np.array([0, 2, 3, 4, 8, 20, 15], dtype=float) # the truth, unobserved
series = 100 + 0.05 * t + weekly[dow] + rng.normal(0, 2, n)
trend = np.convolve(series, np.ones(7) / 7, mode="same") # a centred 7-day mean
resid = series - trend
profile = np.array([resid[dow == d].mean() for d in range(7)])
print(profile.round(2))
print((weekly - weekly.mean()).round(2))
# [-6.46 -4.87 -4.75 -3.73 1.5 13.54 8.63]
# [-7.43 -5.43 -4.43 -3.43 0.57 12.57 7.57]Two ways to hand that to a model. Dummy variables, one per day of week or per month, which are exact and cost a column each — fine for seven days, awkward for 365 days of the year. Or Fourier terms, a handful of sines and cosines at the seasonal frequency, which give a smooth periodic shape in four or six columns and are the standard answer for yearly seasonality on daily data.
import numpy as np
def fourier_terms(t, period, k):
"""2k columns capturing a smooth repeating pattern of the given period."""
out = []
for i in range(1, k + 1):
out.append(np.sin(2 * np.pi * i * t / period))
out.append(np.cos(2 * np.pi * i * t / period))
return np.column_stack(out)
t = np.arange(730)
X_year = fourier_terms(t, period=365.25, k=3) # 6 columns for yearly shape
X_week = fourier_terms(t, period=7, k=2) # 4 columns for weekly shape
print(X_year.shape, X_week.shape) # (730, 6) (730, 4)Multiple seasonalities are normal
Hourly data usually has a daily cycle, a weekly cycle and a yearly one at once, and a model given only the strongest will look like it is failing at random. Add Fourier terms for each period separately; they add rather than interfere. Holidays are not seasonality — they move relative to the calendar — so they need their own indicator features, including the days either side, which often behave more unusually than the holiday itself.
Beat the naive baseline first#
Predicting that tomorrow equals today, or equals the same day last week, is often surprisingly strong. Many forecasting projects produce models that never beat it, and nobody checked. Compute it before anything else.
import numpy as np
def naive(history, horizon):
"""Tomorrow equals today. The hardest baseline on a random walk."""
return np.full(horizon, history[-1])
def seasonal_naive(history, horizon, season=7):
"""Each future day equals the same weekday last week."""
last = history[-season:]
return np.array([last[i % season] for i in range(horizon)])
def drift(history, horizon):
"""Extend the straight line from the first to the last observation."""
slope = (history[-1] - history[0]) / (len(history) - 1)
return history[-1] + slope * np.arange(1, horizon + 1)
h = np.array([100.0, 104, 99, 103, 108, 130, 120, 102, 106, 101, 105, 110, 132, 122])
print(naive(h, 3).round(1)) # [122. 122. 122.]
print(seasonal_naive(h, 3).round(1)) # [102. 106. 101.]
print(drift(h, 3).round(1)) # [123.7 125.4 127.1]Where people go wrong
A rolling feature that includes the current value. The offline score is superb and the model is useless, because at prediction time that value is exactly what you do not have.
If this comes up in an interview
Why can you not use ordinary cross-validation on a time series?
It shuffles, so future data ends up in training and past data in validation. Because consecutive points are correlated, the model effectively sees the answer and the score is excellent and meaningless. Use a forward-chaining split instead.
What is the first thing you do on a forecasting problem?
Compute the naive baseline: tomorrow equals today, or equals the same day last week. It is often hard to beat, and many forecasting projects never do without realising.
How do you build a rolling average feature safely?
Shift before rolling, so the window covers only past values. A rolling mean that includes the current row leaks the target and produces a model that cannot work in production.
Build this: Build a forecast with a naive baseline, a shuffled cross-validation score and a time-aware one. The gap between the last two is the size of the leak you would have shipped.
Watch it explained
This is one section of Time series, 17 hours across 4 sections. Read it in full to get the whole topic in order.