Chapter 6 of 8

Classification and the Economics of Evaluation

3 sections · ~9 min read · 3 practice questions · part of Business Analytics

Learning objectives

  • Build a confusion matrix and compute precision, recall, specificity, and F1 correctly
  • Explain why accuracy is misleading for rare events and choose a metric that matches the cost structure
  • Set a decision threshold by marginal economics rather than by convention

The confusion matrix and the four numbers that matter

A classifier assigns each case to a predicted class, and comparing predictions against outcomes gives four counts: true positives, false positives, false negatives, and true negatives. Every classification metric is a ratio of these four. Precision, true positives over all predicted positives, answers the question the operations team asks: of the cases we act on, what fraction were worth acting on. Recall, also called sensitivity, is true positives over all actual positives, and answers the question the risk team asks: of the cases that mattered, what fraction did we catch. Specificity is true negatives over all actual negatives, and matters when the cost of disturbing an innocent case is high. F1 is the harmonic mean of precision and recall, a single summary that punishes imbalance between them, and it is a reasonable default only when false positives and false negatives cost about the same. Precision and recall trade off against each other because both move with the threshold: lowering the threshold catches more true positives while sweeping in more false ones. Quoting one without the other is the oldest trick in model marketing, since any recall figure up to one hundred percent is achievable by flagging everybody.

Why accuracy fails on rare events

Accuracy is the fraction of all cases classified correctly, and on imbalanced problems it is close to useless. If six percent of customers churn in a given month, a model that predicts nobody churns is ninety-four percent accurate while providing no information and supporting no action. Any genuinely useful churn model that flags real cases will accept some false positives and may score lower on accuracy than the do-nothing baseline. This is the accuracy paradox, and it appears wherever the interesting class is rare, which describes fraud, churn, equipment failure, safety incidents, and most of the problems worth modelling. The fix is to compare against the right baseline and to use metrics that condition on the class. Always report the prevalence alongside any accuracy figure, and always state what the majority-class baseline would score. When a single number is needed for model selection, the area under the precision-recall curve is usually more informative than the area under the ROC curve on heavily imbalanced data, because the ROC curve's specificity term is dominated by a huge true-negative count and moves very little as the threshold changes.

The threshold is a business decision, not a default

The value 0.5 is a convention with no economic content. The right threshold follows from the payoff of acting. Suppose contacting a customer with a retention offer costs a fixed amount, the offer succeeds with some probability, and a successful save is worth the retained margin. Acting on a customer is worthwhile when their probability of churning, times the success rate, times the value of a save, at least covers the cost of contact. Rearranging gives a breakeven probability, and the correct policy is to contact everyone above it. In practice this must be applied to the marginal group rather than the average. Lowering a threshold adds a block of customers whose precision is lower than the precision of everyone above; the right question is whether the precision of that added block, not the overall precision, clears breakeven. There is one large caveat. Model scores are not probabilities unless the model has been calibrated, and many algorithms produce scores that rank well but are badly scaled. Before applying a probability-based breakeven rule to a raw score, check calibration by bucketing scores and comparing the mean predicted score in each bucket with the observed rate, or simply evaluate candidate thresholds directly on their realised net value.

Worked example

Problem

A hypothetical subscription business scores 10,000 customers for churn risk in a month in which 600 actually churn. At a score threshold of 0.50 the model flags 900 customers, of whom 420 actually churn. A retention offer costs 40 dollars per contacted customer, saves a would-be churner with probability 0.35, and a saved customer is worth 600 dollars in retained contribution. Two alternative thresholds are available: at 0.30 the model flags 2,000 customers of whom 540 actually churn; at 0.70 it flags 400 of whom 260 actually churn. Compute the classification metrics at 0.50 and choose the threshold that maximises net value.

Step by step

  1. Confusion matrix at 0.50. True positives = 420. False positives = 900 - 420 = 480. False negatives = 600 - 420 = 180. True negatives = 10,000 - 900 - 180 = 8,920. Check the total: 420 + 480 + 180 + 8,920 = 10,000, and 420 + 180 = 600 actual churners, both correct.
  2. Precision = 420 / 900 = 0.4667, so 46.7 percent of contacted customers were genuinely at risk.
  3. Recall = 420 / 600 = 0.7000, so the model caught 70 percent of churners.
  4. Specificity = 8,920 / 9,400 = 0.9489.
  5. F1 = 2 x (0.4667 x 0.7000) / (0.4667 + 0.7000) = 2 x 0.32667 / 1.16667 = 0.65333 / 1.16667 = 0.5600.
  6. Accuracy = (420 + 8,920) / 10,000 = 0.9340. The majority-class baseline of predicting that nobody churns scores 9,400 / 10,000 = 0.9400, which is higher, so accuracy would rank the useful model below the useless one.
  7. Net value at 0.50. Contact cost = 900 x 40 = 36,000 dollars. Saves = 420 x 0.35 = 147 customers. Value of saves = 147 x 600 = 88,200 dollars. Net = 88,200 - 36,000 = 52,200 dollars.
  8. Net value at 0.30. Contact cost = 2,000 x 40 = 80,000 dollars. Saves = 540 x 0.35 = 189. Value = 189 x 600 = 113,400 dollars. Net = 113,400 - 80,000 = 33,400 dollars.
  9. Net value at 0.70. Contact cost = 400 x 40 = 16,000 dollars. Saves = 260 x 0.35 = 91. Value = 91 x 600 = 54,600 dollars. Net = 54,600 - 16,000 = 38,600 dollars.
  10. Breakeven probability of churn for contacting one customer: p x 0.35 x 600 must be at least 40, so p is at least 40 / 210 = 0.1905, about 19 percent.
  11. Marginal check, 0.70 down to 0.50. Extra contacts = 900 - 400 = 500, extra true positives = 420 - 260 = 160, so marginal precision = 160 / 500 = 0.3200, which exceeds the 0.1905 breakeven. Lowering to 0.50 is correct.
  12. Marginal check, 0.50 down to 0.30. Extra contacts = 2,000 - 900 = 1,100, extra true positives = 540 - 420 = 120, so marginal precision = 120 / 1,100 = 0.1091, which is below the 0.1905 breakeven. Lowering to 0.30 destroys value, and the loss is 1,100 x (0.1091 x 210 - 40) = 1,100 x (22.91 - 40) = -18,800 dollars, matching 52,200 - 33,400 = 18,800.

Answer. At the 0.50 threshold: precision 0.467, recall 0.700, specificity 0.949, F1 0.560, accuracy 0.934 against a do-nothing baseline of 0.940. Net value is 52,200 dollars per month at threshold 0.50, versus 38,600 at 0.70 and 33,400 at 0.30, so 0.50 is the best of the three. The marginal analysis explains why and gives the general rule: keep lowering the threshold while the precision of the newly added block exceeds 19 percent, and stop when it does not. Note that the score of 0.30 does not mean a 30 percent churn probability; the block added between 0.30 and 0.50 had a realised churn rate of only 10.9 percent, so the score is not calibrated and the breakeven rule must be applied to observed marginal precision rather than to the score itself. All figures are for one month, ignore any effect of the offer on customers who were never going to churn, and assume the 0.35 save rate is constant across risk levels, which is itself worth testing.

Practice

Work each question before opening the solution.

  1. A vendor advertises a fraud model with 99.2 percent accuracy on a portfolio where 0.5 percent of transactions are fraudulent. What is the first question to ask, and why?

    Show solution for question 1

    Ask for the confusion matrix, or at minimum precision and recall at the operating threshold. A model that flags nothing at all is 99.5 percent accurate on this portfolio, so 99.2 percent accuracy is worse than doing nothing and is consistent with a model that catches very little fraud while raising some false alarms. Accuracy is dominated by the true negative count when the positive class is rare, so it cannot distinguish a good model from an inert one.

  2. A quality inspection model is evaluated on 5,000 units of which 250 are defective. It flags 300 units and correctly identifies 180 defects. Compute precision, recall, and F1, and describe the operational trade-off.

    Show solution for question 2

    True positives 180, false positives 300 - 180 = 120, false negatives 250 - 180 = 70, true negatives 5,000 - 300 - 70 = 4,630. Precision = 180 / 300 = 0.600. Recall = 180 / 250 = 0.720. F1 = 2 x (0.600 x 0.720) / (0.600 + 0.720) = 2 x 0.432 / 1.320 = 0.864 / 1.320 = 0.6545. Operationally, 40 percent of flagged units waste inspection time, while 70 defects per 5,000 reach customers. Which error to reduce depends on the cost of an inspection versus the cost of a defect escaping, including warranty and reputational cost.

  3. A model scores 0.91 area under the ROC curve in development and performs poorly in production three months later. Give two distinct explanations and a test for each.

    Show solution for question 3

    First, leakage: a feature available at training time encoded information not available at the moment of prediction, for example a cancellation reason code that is only populated after the outcome. Test by re-training with a strict as-of cut, using only features whose values were knowable before the prediction timestamp, and confirm the score drops to a plausible level. Second, drift: the population or the relationship changed, through a pricing change, a new acquisition channel, or a seasonal shift. Test by comparing the distribution of each feature and of the model score between the training window and recent production data, and by scoring a fresh labelled sample from the current period.