Forecasting Rolling origin Runs in your browser

Forecast Decision Lab

Three ordinary forecasting methods, judged the way a planner would actually experience them: refit at every origin, forecast forward, compare against what happened next, and never once look at a number the forecaster could not have seen. Then price the errors, because being wrong high and being wrong low almost never cost the same.

How the evaluation works

Rolling origin

A single train-test split gives you one number from one arbitrary cut point. Rolling origin walks the cut point forward instead. At origin t the lab hands each method only the values before t, takes its forecast for the next h periods, and scores it against what actually followed. Then the origin advances one period and the whole thing repeats. You end up with many forecasts made under many different histories, which is a much steadier read than one lucky or unlucky split.

Where leakage would creep in, and what stops it

Leakage is when a forecast is quietly built using information from after the moment it claims to be made. It is easy to introduce by accident and it makes every method look better than it is. Three places it would happen here, and how each is closed:

The smoothing constant. Fitting one alpha to the whole series and then evaluating with it is leakage, because the early forecasts were tuned on data from the end. If you turn on refitting, alpha is re-chosen at every origin using only that origin's training window.
The trend line. The regression is refit from scratch at every origin on the training window alone, and then extrapolated forward. It never sees the points it is about to be scored against.
Scaling and centring. There is none. Nothing is normalised using series-wide means or ranges, which is one of the most common quiet leaks in forecasting code.

The self-tests below check this directly rather than taking it on trust: they overwrite the tail of the series with absurd numbers and confirm that every forecast made before that point comes back bit for bit identical.

The three methods

Moving average of the last k periods. A flat forecast. No trend, no seasonality, no memory beyond the window. It is the honest baseline that anything more elaborate should have to beat.
Simple exponential smoothing. The level is updated as level = alpha × latest + (1 − alpha) × previous level, started at the first observed value, and the forecast is that level held flat. A high alpha chases the most recent point; a low alpha barely moves.
Linear trend. Ordinary least squares of the value on the period index over the training window, projected forward. It is the only one of the three that can follow a drift, and the only one that can run away badly when the drift stops.

None of the three handles seasonality. If you build a scenario with a strong seasonal swing, expect all three to do poorly in a patterned way — that is the point of having the option.

Scoring on accuracy and on cost

Errors are defined as forecast − actual, so a positive error is an over-forecast. MAE and RMSE both treat the two directions as equally bad. The operational cost does not: over-forecasting is charged one rate per unit and under-forecasting another. In most supply chains those two numbers are nowhere near equal, and the method with the best RMSE is not always the method with the lowest cost. Watching those two rankings disagree is the most useful thing this lab does.

Set up the scenario

The series

Where the data comes from: Generate a seeded scenario Paste my own numbers

Periods

Starting level

Drift per period

Seasonal swing (amplitude)

Season length

Noise standard deviation

Level shift at period

Set this to zero for no shift. A sudden change in demand can leave a forecast based on older observations behind.

Size of that shift

Seed

Evaluation

First origin after period

How much history the first forecast is allowed to use.

Forecast horizon h

Periods ahead, forecast from each origin.

Moving average window k

Smoothing constant alpha

Fixed alpha

Between 0 and 1. Ignored when refitting is on.

What errors cost

Over-forecasting, per unit

What a unit of surplus costs: holding, markdown, obsolescence.

Under-forecasting, per unit

What a unit of shortfall costs: lost margin, expediting, a missed promise.

Constant adjustment applied to every forecast

Added to each forecast before scoring. Leave at zero to score the raw methods.

Results

Every method scored over the same set of origins. The shaded cell in each column is the best method by that measure.
Method	MAE	RMSE	Bias (mean error)	Over-forecast cost	Under-forecast cost	Total cost	Cost per period

The cost-minimising adjustment

Because the two error directions are priced differently, the cheapest forecast is not the most accurate one — it is one deliberately biased toward the expensive side. The table below reports the constant shift that would have minimised cost on this evaluation window for each method.

These numbers are fitted with hindsight and will flatter themselves. The adjustment is chosen by looking at the very errors it is then scored on, so the saving shown is an optimistic ceiling, not what you would get next quarter. Treat it as a measure of how much asymmetry is on the table, and if you want an honest estimate, pick the shift on early origins and check it on later ones.

The best constant shift on this window, and what it would have saved.
Method	Best shift	Cost at that shift	Cost as scored	Hindsight saving

The actual series with each method's forecast for the first step of each origin. The shaded block on the left is history used only for training, before the first origin.

Actual Moving average Exponential smoothing Linear trend

Forecasts and errors, origin by origin

Every forecast the evaluation made, with the actual value and the resulting error and cost.
Method	Origin	Step	Target period	Forecast	Actual	Error	Cost

Accuracy by how far ahead

Forecasting further out is harder. This breaks the same errors down by step.

MAE and RMSE at each step of the horizon.
Method	Step	Forecasts	MAE	RMSE	Total cost

Export

Every export repeats the scenario, the seed, and the evaluation settings.

Self-tests

The leakage checks in here are the ones worth watching. They take a series, run the evaluation, then rewrite every value from some period onward with wildly different numbers and run it again. Any forecast made at or before that period must come back identical to the last decimal place; if it moves, the method peeked. The rest of the checks pin the arithmetic to values that can be worked out by hand.

Also callable from the console as window.runToolTests().

Things worth trying

Set the two cost rates far apart — say 1 against 20 — and watch the cost ranking pull away from the RMSE ranking. The most accurate method stops being the cheapest one.
Put a level shift halfway through. The linear trend usually suffers worst, because it keeps projecting the old slope from a level that no longer exists, and the moving average recovers fastest.
Turn the drift off and leave the trend method in. On a flat series the fitted slope is noise, and extrapolating noise is worse than forecasting nothing at all.
Switch alpha from fixed to refit. On a stable series it barely helps. On a series with a shift it helps a lot, because the refit lets alpha climb once the old level stops being useful.
Add a strong seasonal swing. All three methods degrade, and the error pattern repeats on the season length — which is exactly the diagnostic that tells you the model is missing a term.