Shelly guideForecasts: how good are they?
Forecast accuracy is measured, not assumed. The same approach as Pavi's MAB thesis, where seasonal naive beat XGBoost.
At a glance
- Rolling-origin backtest: 4 folds × 7 days
- Scored by WAPE (total absolute error ÷ total actual) and MAPE
- Best model chosen per category
- Cross-checked independently in R
How it works
- SARIMA-X: order (1,0,1)(1,1,1)[7] with a NZ public-holiday regressor, fitted on the last 364 days. If it becomes numerically unstable, it falls back to seasonal naive.
- XGBoost: one global model across categories with weekday, day-of-year, holiday and lagged features (lags ≥ 7, so a 7-day forecast needs no recursion).
- Baselines: last value, same day last week, and drift.
How to read it
- If the best model only beats seasonal naive by a point or two, the extra complexity isn't worth much.
- A sudden jump in a model's error is flagged by the learning loop as possible drift.
Limits & safeguards
- No weather or promo calendar yet. Both are on the roadmap as model inputs.
Ask Shelly
Which model is most accurate?Forecast for next week
Type or say these in the chat on the main page.