Chapter 2 of 8

Forecast Validation and Error Diagnosis

3 sections · ~8 min read · 3 practice questions · part of Supply Chain Analytics

Learning objectives

  • Design an honest holdout test that does not leak future information into the model
  • Compute and interpret MAE, RMSE, MAPE, mean error, and a scaled error ratio against a naive baseline
  • Separate bias from dispersion and choose the corresponding corrective action

Validation is a design problem before it is an arithmetic problem

A close fit to history does not establish forecast accuracy: a flexible model may fit noise as well as useful patterns. The only meaningful test is out-of-sample: fit on data the model is allowed to see, then score on data it has never seen. In time series this must respect chronology. Randomly shuffling weeks into training and test sets leaks future information backward and produces error statistics that are far too optimistic. The correct structure is a rolling origin: fit through week t, forecast weeks t plus one onward at the lead time you actually care about, advance the origin, repeat. The lead time matters enormously. A one-week-ahead forecast and a six-week-ahead forecast are different problems with different error magnitudes, and a validation that mixes horizons produces a number that describes neither. Fix the horizon to the decision it feeds. If the replenishment lead time is five weeks, validate at a five-week horizon, because that is the uncertainty the safety stock has to absorb. Finally, hold out enough periods that the score is not dominated by one unusual week, and record the holdout window explicitly so results are reproducible.

What each error statistic is actually measuring

Define the error as actual minus forecast, so a positive error means the forecast was too low. Mean error, also called bias, averages the signed errors and detects a systematic tilt; its units are the units of demand. Mean absolute error averages the magnitudes and is in the same units, which makes it directly interpretable as typical miss size. Root mean squared error squares the errors before averaging, which penalizes large misses far more heavily, so RMSE always equals or exceeds MAE and the gap between them widens as the error distribution gets more spread out or more prone to occasional large misses. Mean absolute percentage error normalizes by the actual and is dimensionless, which permits comparison across items of different volume, but it is unstable when actuals approach zero and it penalizes over-forecasting differently from under-forecasting. Mean percentage error is the signed version and is the percentage form of bias. None of these tells you whether the forecast is any good in absolute terms, because a MAPE of four percent is excellent for volatile demand and mediocre for a flat series.

Baselines, bias, and what to do about each

The fix for interpretability is a baseline. Compute the same error statistic for a naive forecast, typically last period's actual carried forward, over the identical holdout window, then take the ratio of the model MAE to the naive MAE. A ratio below one means the model beats the naive rule; a ratio at or above one means the model is not earning its maintenance cost. This scaled ratio is comparable across items and horizons in a way that raw MAE is not. Once the model has passed the baseline test, diagnose bias and dispersion separately, because they have different remedies. Bias, a non-zero mean error, is usually a process problem: a sales team that consistently sandbags, a promotion not represented in the model, or a level shift the model has not caught up to. Bias is cheap to fix, often by simply adding the estimated bias back into the forecast, and correcting it lowers MAE and RMSE at the same time. Dispersion, a large MAE with near-zero bias, is genuine uncertainty and cannot be removed by arithmetic. Dispersion is what safety stock exists to absorb, so the correct response is to size the buffer against it rather than to keep tuning the model.

Worked example

Problem

Marlow Instruments validates a weekly demand model for one item on an eight-week holdout. Actual demand in units for weeks 1 through 8 was 420, 385, 460, 500, 445, 390, 505, 470. The model forecast 400, 400, 450, 470, 460, 410, 480, 455. Demand in the week immediately before the holdout was 410 units. Compute mean error, MAE, RMSE, MAPE, and MPE for the model; compute MAE for a naive last-value-carried-forward baseline over the same eight weeks; report the scaled error ratio; then compute the effect of removing the bias.

Step by step

  1. Errors (actual minus forecast) week by week: 420-400 = 20, 385-400 = -15, 460-450 = 10, 500-470 = 30, 445-460 = -15, 390-410 = -20, 505-480 = 25, 470-455 = 15.
  2. Sum of errors = 20 - 15 + 10 + 30 - 15 - 20 + 25 + 15 = 50 units. Mean error (bias) = 50 / 8 = 6.25 units, so the model runs low by about 6.25 units per week on average.
  3. Sum of absolute errors = 20 + 15 + 10 + 30 + 15 + 20 + 25 + 15 = 150 units. MAE = 150 / 8 = 18.75 units.
  4. Sum of squared errors = 400 + 225 + 100 + 900 + 225 + 400 + 625 + 225 = 3,100. Mean squared error = 3,100 / 8 = 387.5. RMSE = square root of 387.5 = 19.685 units.
  5. Absolute percentage errors: 20/420 = 4.762 percent, 15/385 = 3.896, 10/460 = 2.174, 30/500 = 6.000, 15/445 = 3.371, 20/390 = 5.128, 25/505 = 4.950, 15/470 = 3.191. Sum = 33.472 percent. MAPE = 33.472 / 8 = 4.184 percent.
  6. Signed percentage errors sum to 4.762 - 3.896 + 2.174 + 6.000 - 3.371 - 5.128 + 4.950 + 3.191 = 8.682 percent. MPE = 8.682 / 8 = 1.085 percent, confirming a small low-side tilt.
  7. Naive baseline forecasts each week with the prior week's actual: 410, 420, 385, 460, 500, 445, 390, 505. Naive errors: 10, -35, 75, 40, -55, -55, 115, -35. Sum of absolute naive errors = 10 + 35 + 75 + 40 + 55 + 55 + 115 + 35 = 420. Naive MAE = 420 / 8 = 52.50 units.
  8. Scaled error ratio = model MAE / naive MAE = 18.75 / 52.50 = 0.3571.
  9. Bias removal: add the estimated bias of 6.25 units to every forecast, giving 406.25, 406.25, 456.25, 476.25, 466.25, 416.25, 486.25, 461.25. New errors are 13.75, -21.25, 3.75, 23.75, -21.25, -26.25, 18.75, 8.75. New mean error = 0.00 by construction. New MAE = 137.50 / 8 = 17.19 units. New RMSE = 18.67 units. New MAPE = 3.93 percent.

Answer. Model MAE = 18.75 units, RMSE = 19.685 units, MAPE = 4.18 percent, bias = +6.25 units per week (about 1.4 percent of the mean actual of 446.875 units). The naive baseline MAE is 52.50 units, so the scaled ratio of 0.357 means the model cuts typical error to roughly 36 percent of the naive rule and is clearly worth keeping. Removing the 6.25-unit bias lowers MAE to 17.19 units, RMSE to 18.67 units, and MAPE to 3.93 percent, an 8.3 percent MAE improvement from arithmetic alone. The RMSE-to-MAE ratio of 19.685 / 18.75 = 1.05 is close to one, indicating no single catastrophic miss dominates; the residual error is broad dispersion, which is the safety stock's job rather than the model's.

Practice

Work each question before opening the solution.

  1. A colleague proposes judging the model by MAPE alone and setting a target of under 5 percent for every item in the catalogue. Give two specific reasons this target is unsound.

    Show solution for question 1

    First, MAPE has no absolute meaning without a baseline: for a nearly flat series a naive rule might achieve 2 percent, so a 4.18 percent model would be worse than doing nothing, while for a highly volatile item 15 percent might be excellent. Second, MAPE is unstable and can be enormous or undefined for slow-moving items where weekly actuals are small or zero, so a uniform percentage target penalizes exactly the items where percentage error is least meaningful.

  2. Suppose the same model is scored on a different item and returns MAE = 61 units against a naive MAE of 58 units, with a bias of -2 units. What should the planner do, and why?

    Show solution for question 2

    The scaled ratio is 61 / 58 = 1.052, so the model performs slightly worse than carrying last week's actual forward, and the near-zero bias means there is no easy correction available. The planner should retire the model for this item, revert to the naive rule or a simple smoothing method, and size safety stock against the observed dispersion rather than spending further effort tuning a model that does not beat its baseline.

  3. The replenishment lead time for this item is five weeks, but the validation above used a one-week horizon. What is the practical consequence, and how should the validation be redesigned?

    Show solution for question 3

    One-week errors are systematically smaller than five-week errors, so using MAE = 18.75 units to size a five-week buffer will understate the required safety stock and cause stockouts. The validation should be rerun on a rolling origin that forecasts five weeks ahead and scores the error of the cumulative five-week demand, since that cumulative quantity is what the reorder point must cover.