Chapter 1 of 4

Choosing and Validating a Forecast Model

3 sections · ~9 min read · 3 practice questions · part of SC Analytics & Technology

Learning objectives

  • Design a holdout backtest that gives an honest estimate of forward accuracy rather than in-sample fit.
  • Benchmark a candidate model against a naive forecast using MASE and interpret the result.
  • Use a tracking signal to detect a live forecast that has drifted off-centre.

In-Sample Fit Is Not Accuracy

The Demand & Inventory Planning course covers how the common forecasting methods work and how the error metrics are defined. This chapter takes the next step and asks a harder question: how do you know the model you have chosen is any good? The trap that catches most teams is evaluating a model on the same data used to fit it. Every additional parameter, every tuned smoothing constant, every seasonal index buys a little more in-sample fit, and almost none of that gain survives contact with next month's demand. A model tuned to minimise in-sample MAPE will reliably look better than a simple one and will reliably disappoint. The fix is structural rather than statistical. Split the history into a training window and a holdout window, fit only on the training window, and score only on the holdout. The holdout must sit at the end of the series, not be sampled randomly out of the middle, because demand is a time series and randomly sampled validation lets the model see the future. For a rolling operation the stronger version is a rolling-origin backtest: fit through week t, forecast week t plus one, advance, and repeat, so that every scored forecast was produced with only the information a planner would have had at the time. This is more work to set up and it is the only design that gives a number you can defend. State the holdout length alongside the accuracy figure, because a MAPE quoted without a validation design is not a measurement.

Benchmarking Against Naive: MASE

A forecast accuracy number is meaningless in isolation. A MAPE of 8 percent sounds excellent until you learn that simply repeating last week's actual would have produced 7 percent. The discipline is to always score a benchmark alongside the candidate, and the right default benchmark is the naive forecast: next period equals this period. Mean absolute scaled error (MASE) formalises the comparison. Compute the mean absolute error of the candidate on the holdout, then divide by the mean absolute error the naive method achieved on the training data. A MASE below 1 means the holdout MAE is below the training-sample naive scaling error; a direct performance comparison still requires both forecasts on the same holdout. The scaling makes MASE comparable across items of wildly different volume, which is exactly what a portfolio review needs and what MAD cannot provide. It is also defined when actuals are zero, which is where MAPE fails. Read the magnitude, not just the sign. A MASE of 0.97 means the candidate MAE is three percent below the training-sample scaling error. To compare the candidate with a naive forecast, score both on the same holdout. Whether an improvement is worth adopting depends on its stability and the decision savings relative to implementation and operating cost; a MASE near one alone does not establish that demand follows a random walk.

Monitoring a Live Forecast: The Tracking Signal

Validation happens once; monitoring happens forever. A model that passed its backtest can still go wrong in production, because the business changes: a competitor exits, a promotion calendar shifts, a product cannibalises its predecessor. The cheapest detector is the tracking signal, which is the running sum of signed forecast errors divided by the mean absolute deviation of those errors. Random errors cancel, so a healthy forecast keeps the running sum near zero and the tracking signal oscillates around it. A signal that walks steadily away from zero means the errors share a sign, which is the definition of bias and usually means the model is missing a trend or a level shift. A common control limit is plus or minus four; crossing it should trigger a review rather than an automatic refit, because the right response depends on whether the cause is a genuine structural break or a one-off event that should be excluded from history. One useful property is worth internalising: if every error in the window has the same sign, the tracking signal equals the number of periods observed. So four consecutive under-forecasts alone put the signal at exactly the conventional limit. That is deliberately sensitive, and it should be, because bias compounds into inventory in a way that noise does not.

Worked example

Problem

Twelve weeks of demand for one SKU: 320, 340, 310, 360, 350, 340, 380, 360, 400, 370, 410, 390. Fit simple exponential smoothing with alpha = 0.30 on weeks 1 to 8 only, initialising F1 = 320, then produce one-step-ahead forecasts for the weeks 9 to 12 holdout. Compute (a) the candidate's MAE on the holdout, (b) the naive method's MAE on the same holdout, (c) MASE for both, and (d) the tracking signal at the end of the holdout. Then say whether the model should be adopted.

Step by step

  1. Update rule: F(t+1) = F(t) + 0.30 x (A(t) - F(t)).
  2. Training window rolls forward: F2 = 320.00, F3 = 326.00, F4 = 321.20, F5 = 332.84, F6 = 337.99, F7 = 338.59, F8 = 351.01.
  3. Holdout forecasts, each using only data through the prior week: F9 = 351.01 + 0.30 x (360 - 351.01) = 353.71. F10 = 353.71 + 0.30 x (400 - 353.71) = 367.60. F11 = 367.60 + 0.30 x (370 - 367.60) = 368.32. F12 = 368.32 + 0.30 x (410 - 368.32) = 380.82.
  4. Holdout errors (actual minus forecast): week 9 = 400 - 353.71 = +46.29; week 10 = 370 - 367.60 = +2.40; week 11 = 410 - 368.32 = +41.68; week 12 = 390 - 380.82 = +9.18.
  5. Candidate MAE = (46.29 + 2.40 + 41.68 + 9.18) / 4 = 99.55 / 4 = 24.89 units.
  6. Naive forecasts on the holdout are the prior actuals: 360, 400, 370, 410. Errors: +40, -30, +40, -20. Naive MAE = (40 + 30 + 40 + 20) / 4 = 130 / 4 = 32.50 units.
  7. MASE scale factor is the naive method's MAE on the training window. Training one-step naive errors, weeks 2 to 8: 20, 30, 50, 10, 10, 40, 20. Sum = 180 over 7 observations = 25.71 units.
  8. MASE candidate = 24.89 / 25.71 = 0.97. MASE naive on the holdout = 32.50 / 25.71 = 1.26.
  9. Tracking signal: running sum of signed errors = +46.29 + 2.40 + 41.68 + 9.18 = +99.55. Divided by the holdout MAD of 24.89 gives a tracking signal of 4.00.
  10. Read the two results together. MASE of 0.97 says the model beat naive by 3 percent. The tracking signal of 4.00 says every single holdout error was an under-forecast, which is what happens when a smoothing model with no trend term is fed a series that is climbing roughly 7 units a week.

Answer. Candidate MAE = 24.89 units, naive MAE = 32.50 units, MASE = 0.97 against 1.26 for naive, tracking signal = +4.00. The candidate improves holdout MAE by about 23 percent relative to naive, but the short holdout and tracking signal call for more validation before adoption: the series has a trend and simple exponential smoothing structurally cannot follow one, so it under-forecasts every period and the errors never cancel. The right next step is a method with a trend term, such as Holt's linear method, re-validated on the same holdout. Note also the limitation of a four-week holdout: it is short enough that one unusual week would move MAE noticeably, so treat these figures as directional and prefer a rolling-origin backtest over more history before committing.

Practice

Work each question before opening the solution.

  1. A planner tunes alpha to minimise error across the full 12-week series and reports a much lower MAE than the backtest above. Why is that number not comparable, and what has actually been measured?

    Show solution for question 1

    Tuning alpha on the full series means the parameter was chosen with knowledge of weeks 9 to 12, so those weeks are no longer a holdout and the reported figure is in-sample fit, not forward accuracy. What has been measured is how well the model can be bent to describe history it has already seen, which always improves as tuning effort rises and tells you nothing about next week. The backtest number is the defensible one because every forecast in it was produced from information available before the actual arrived. If alpha genuinely needs tuning, tune it on the training window alone and leave the holdout untouched.

  2. A second model is tested on the same holdout and posts an MAE of 18.0 units. Compute its MASE using the scale factor above, and say what would still need checking before adopting it.

    Show solution for question 2

    MASE = 18.0 / 25.71 = 0.70, a 30 percent improvement over naive and a clear gain against the first model's 0.97. That is worth taking seriously, but two checks remain. First, the tracking signal on the holdout: a model can post a good MAE while still being biased, and bias is the error type that turns into surplus or missing inventory. Second, the holdout is only four weeks, so the comparison rests on four observations; re-run both models under a rolling-origin backtest across more history before rebuilding the replenishment policy around the winner.

  3. A live forecast has held a MASE near 0.75 for a year, then the tracking signal climbs from 0 to +5 over six weeks while MASE stays roughly unchanged. What has happened, and why is refitting on all history the wrong first move?

    Show solution for question 3

    A persistently positive tracking signal with stable MASE means the error magnitude has not grown but the errors have all turned the same way: the forecast is now systematically low. Something shifted in the level of demand, from a new customer, a competitor exit, or a price change, rather than the series simply becoming noisier. Refitting on all history is the wrong first move because it averages the new level in with years of the old one, so the model chases the shift slowly and stays biased for months. Establish the cause first. If it is a genuine level shift, reset the level or shorten the history the model sees; if it is a one-off event, exclude those weeks from the fit and keep the existing model.