Asked about Forecasting in code: baselines, backtests and intervals
How would you evaluate a demand forecasting model?
An answer that works
Rolling-origin backtesting that trains only on the past and forecasts the real horizon, a seasonal naive baseline, a scale-free metric such as MASE so products of different volume compare, and interval coverage if the business uses ranges.
Then they push further
Your model beats the baseline on average but loses badly on promotions. What next?
Promotions are a known future event, so add them as features, including planned ones, and evaluate that slice separately. If data is thin, a separate adjustment for promotion periods may beat one global model.
Why not use mean absolute percentage error?
It is undefined or huge when actual values are near zero and penalises over- and under-forecasts asymmetrically. Scaled errors avoid both problems.
What gets marked up
Marked up for rolling backtests, a naive baseline, and a scale-free metric. Random train-test splits are marked down immediately.
Reading an answer is not being able to give one. The section that teaches this is Forecasting in code: baselines, backtests and intervals, 5 hours of Time series.
Learn this properlyAlso asked about Time series