Chapter 5 of 8

Regression for Prediction and Interpretation

3 sections · ~9 min read · 3 practice questions · part of Business Analytics

Learning objectives

  • Fit a simple linear regression by hand and interpret the slope in business units
  • Compute the residual standard error, the standard error of the slope, and a confidence interval
  • Separate statistical significance from economic significance and state the limits of extrapolation

What the line is and what the slope means

Simple linear regression fits the straight line that minimises the sum of squared vertical distances between the observed points and the line. The slope equals the sum of the products of the deviations of the two variables from their means, divided by the sum of squared deviations of the predictor. The intercept then follows from the requirement that the line pass through the point of means. Two features fall out of the mathematics and are worth remembering because they double as arithmetic checks: the residuals always sum to zero, and the line always passes through the mean of the predictor and the mean of the response. The slope is the workhorse of interpretation, and its unit is the unit of the response divided by the unit of the predictor. If the response is units sold per week and the predictor is advertising spend in thousands of dollars per week, the slope is additional units sold per additional thousand dollars of weekly spend. Always say it that way in a report. Without units, a reader cannot tell whether the slope describes a useful change in the outcome.

How well does it fit, and how sure are we of the slope

Two different questions get confused here. The coefficient of determination, R squared, is the share of the variation in the response that the line accounts for, computed as one minus the sum of squared residuals over the total sum of squares. It describes fit within the sample and nothing else. A high R squared does not mean the model will predict well out of sample, does not mean the relationship is causal, and rises mechanically as predictors are added. The residual standard error, computed as the square root of the sum of squared residuals divided by the number of observations minus two, is far more useful in practice because it is in the units of the response and answers the question a manager actually asks: how far off is a typical prediction. The uncertainty in the slope is a separate matter again, captured by the standard error of the slope, which is the residual standard error divided by the square root of the sum of squared deviations of the predictor. That formula shows how to buy precision: gather more observations, and spread them wider across the range of the predictor. Two clustered points can never pin down a slope, however clean the data looks.

Significance is not importance, and the line stops where the data stops

A slope can be measured precisely enough to rule out zero and still be far too small to justify the spending it describes. Statistical significance answers whether an effect exists; economic significance answers whether it pays. The test is simple: convert the slope into money using contribution margin, and compare it against the cost of the input. If an extra thousand dollars of advertising reliably buys six extra units, and each unit contributes forty dollars, the thousand dollars returns two hundred and forty dollars and the campaign destroys value even though the coefficient is highly significant. Report both numbers or the reader will assume significance implies profitability. Two further limits belong in every regression writeup. First, the fitted line is evidence only across the range of the predictor observed in the sample; predicting at a spend level three times the highest ever tried is extrapolation, and diminishing returns will bend a real response curve long before the straight line does. Second, a regression on observational data measures association. If high-spend weeks were also promotional weeks, the slope carries the promotion's effect too, and the honest sentence in the report says associated with, not causes.

Worked example

Problem

A hypothetical retailer records six weeks of advertising spend, x, in thousands of dollars per week, and units sold, y, per week. The pairs are (2, 118), (4, 138), (6, 139), (8, 158), (10, 166), (12, 181). Fit the least squares line, compute R squared, the residual standard error, the standard error of the slope, and a 95 percent confidence interval for the slope. Then judge whether the advertising pays if contribution margin is 40 dollars per unit.

Step by step

  1. Means. Sum of x = 2 + 4 + 6 + 8 + 10 + 12 = 42, so mean x = 7 thousand dollars. Sum of y = 118 + 138 + 139 + 158 + 166 + 181 = 900, so mean y = 150 units.
  2. Deviations from the mean of x: -5, -3, -1, 1, 3, 5. Deviations from the mean of y: -32, -12, -11, 8, 16, 31.
  3. Sum of squared x deviations, Sxx = 25 + 9 + 1 + 1 + 9 + 25 = 70.
  4. Sum of cross products, Sxy = (-5)(-32) + (-3)(-12) + (-1)(-11) + (1)(8) + (3)(16) + (5)(31) = 160 + 36 + 11 + 8 + 48 + 155 = 418.
  5. Slope b = Sxy / Sxx = 418 / 70 = 5.9714 units per thousand dollars of weekly spend.
  6. Intercept a = mean y - b x mean x = 150 - 5.9714 x 7 = 150 - 41.8 = 108.2 units per week. (Note 418 / 70 x 7 = 41.8 exactly.)
  7. Fitted line: predicted units = 108.2 + 5.9714 x spend.
  8. Fitted values and residuals. x = 2: 120.14, residual 118 - 120.14 = -2.14. x = 4: 132.09, residual +5.91. x = 6: 144.03, residual -5.03. x = 8: 155.97, residual +2.03. x = 10: 167.91, residual -1.91. x = 12: 179.86, residual +1.14. The residuals sum to -2.14 + 5.91 - 5.03 + 2.03 - 1.91 + 1.14 = 0.00, which is the required check.
  9. Sum of squared residuals, SSE = 4.592 + 34.979 + 25.287 + 4.115 + 3.664 + 1.306 = 73.94.
  10. Total sum of squares, SST = sum of squared y deviations = 1024 + 144 + 121 + 64 + 256 + 961 = 2570.
  11. R squared = 1 - 73.94 / 2570 = 1 - 0.0288 = 0.9712. Independent check by the alternative formula: b x Sxy / SST = 5.9714 x 418 / 2570 = 2496.06 / 2570 = 0.9712, which agrees.
  12. Residual standard error = square root of (SSE / (n - 2)) = square root of (73.94 / 4) = square root of 18.486 = 4.30 units per week.
  13. Standard error of the slope = residual standard error / square root of Sxx = 4.30 / square root of 70 = 4.30 / 8.367 = 0.514 units per thousand dollars.
  14. t statistic for the slope = 5.9714 / 0.514 = 11.62 on 4 degrees of freedom, far beyond the two-sided 5 percent critical value of 2.776, so the slope is distinguishable from zero.
  15. 95 percent confidence interval for the slope = 5.9714 plus or minus 2.776 x 0.514 = 5.9714 plus or minus 1.427, giving 4.54 to 7.40 units per thousand dollars.
  16. Economic test. At the point estimate, one extra thousand dollars of weekly spend is associated with 5.97 extra units, worth 5.97 x 40 = 238.86 dollars of contribution against 1,000 dollars of cost. Even at the optimistic end of the interval, 7.40 units x 40 = 296 dollars, still far below 1,000. Breakeven would require 1,000 / 40 = 25 units per thousand dollars, which is about 3.4 times the upper confidence limit of 7.40.

Answer. The fitted line is predicted weekly units = 108.2 + 5.97 times weekly advertising spend in thousands of dollars, with R squared of 0.971, a residual standard error of 4.30 units per week, and a slope standard error of 0.514, giving a 95 percent confidence interval of 4.54 to 7.40 units per thousand dollars. The relationship is strong and statistically clear, and yet the recommendation is to cut this advertising: at 40 dollars of contribution per unit the spend returns about 239 dollars per thousand dollars invested, and breakeven would need 25 units per thousand, which the data rules out. The valid prediction range is 2 to 12 thousand dollars of weekly spend; the intercept of 108.2 units is not a forecast of zero-spend sales because zero lies outside the observed range. Because spend was not randomly assigned across weeks, the slope is an association and may absorb the effect of anything that moved with spend, such as promotions or seasonality.

Practice

Work each question before opening the solution.

  1. Using the fitted line predicted units = 108.2 + 5.97 times spend, predict weekly units at a spend of 5 thousand dollars and at 30 thousand dollars, and state which prediction you would report.

    Show solution for question 1

    At x = 5: 108.2 + 5.97 x 5 = 108.2 + 29.86 = 138.06, about 138 units per week. At x = 30: 108.2 + 5.97 x 30 = 108.2 + 179.14 = 287.34, about 287 units per week. Report only the first. A spend of 5 lies inside the observed range of 2 to 12, so the residual standard error of 4.30 units gives a reasonable sense of the error. A spend of 30 is two and a half times the highest level ever tried; the straight line assumes the response never bends, whereas real advertising response saturates, so 287 units is an arithmetic result rather than a forecast.

  2. A colleague fits a regression of monthly revenue on the number of salespeople, obtains R squared of 0.93, and concludes that hiring ten more salespeople will raise revenue by ten times the slope. Give two reasons this conclusion may be wrong.

    Show solution for question 2

    First, R squared measures in-sample fit, not causality. Headcount and revenue both tend to grow with company size and market conditions, so the slope likely absorbs an omitted common cause and overstates what an incremental hire delivers. Second, the projection assumes a constant slope over a range the data may not cover, and it assumes the territory can absorb ten more people; if the existing headcount range in the data is, say, 12 to 20, a jump to 30 is extrapolation into likely diminishing returns as new hires split existing territories.

  3. Two studies estimate the same slope of 3.0. Study A has a standard error of 0.4, study B has 2.5. Both use the same number of observations. What most likely differs, and which slope would you use for planning?

    Show solution for question 3

    The standard error of a slope is the residual standard error divided by the square root of the sum of squared deviations of the predictor. With the same sample size, study A either has less residual noise or, more commonly, spread its observations more widely across the predictor's range, while study B clustered its observations in a narrow band. Use study A's estimate: its 95 percent interval is roughly 3.0 plus or minus 2 times 0.4, or about 2.2 to 3.8, whereas study B's spans roughly minus 2 to 8 and cannot even rule out a slope of zero.