Twenty tests, one α.
Paste a column of p-values and see what Bonferroni, Holm and Benjamini–Hochberg each do to it. The three disagree because they are protecting against different things, and the table below makes the disagreement concrete rather than abstract.
1 · Your p-values
One test per line. A label, then the p-value, separated by a comma, a tab, an equals sign or just a space. Lines beginning with # are ignored.
2 · Adjusted values, side by side
Every row keeps its input position. Sorting happens inside the procedures, not in this table, so a row’s identity never depends on how the file happened to be ordered.
| Test | Rankby p | punadjusted | BonferroniFWER | HolmFWER | BHFDR | BYFDR, any dependence |
|---|
A tick marks an adjusted value at or below α. Adjusted values are capped at 1, as in R’s p.adjust.
3 · The sorted picture
Sort the p-values smallest to largest and plot each against its rank. Every procedure here is a rule about where those points sit relative to a line.
4 · Two different promises
Bonferroni and Holm control one thing; Benjamini–Hochberg controls another. Neither is the careful choice — they answer different questions, and the right one depends on what a false positive costs you.
Family-wise error rate
FWER = P(at least one false rejection)Hold this at 0.05 and, across the whole family of tests, the chance of making even a single false claim is at most one in twenty. It is a promise about the entire set at once, and it does not soften as the family grows: twenty tests or two thousand, the guarantee is the same, so the price per test keeps rising.
That is the right promise when one false positive is expensive on its own. A confirmatory trial with several endpoints, a regulatory submission, a claim that will be published as a finding rather than as a lead — in all of those, a single wrong rejection is the failure you are insuring against.
Bonferroni multiplies each p-value by the number of tests. It is valid whatever the dependence between tests, requires nothing of them, and is the most conservative option on this page. Holm reaches the same guarantee by working from the smallest p-value upward, and it never rejects less than Bonferroni does. There is no situation in which Bonferroni is preferable to Holm on statistical grounds; Bonferroni survives because it is easy to explain and easy to do in your head.
False discovery rate
FDR = E[ false rejections / total rejections ]Hold this at 0.05 and, on average, no more than one in twenty of the tests you call significant is a false positive. It says nothing about whether you made any errors — only about the expected proportion among your claims. Reject nothing and the ratio is defined as zero.
That is the right promise when the output is a shortlist rather than a verdict. Screening a hundred suppliers, a thousand SKUs, a panel of markers: you expect to follow up on whatever survives, so a few wrong entries in the shortlist cost you some wasted follow-up, not a wrong conclusion.
The gain in power is real and it grows with the family. In the worked example below, ten tests at α = 0.05 give two rejections under Holm and four under BH — the same data, the same α, a different promise.
Bonferroni and Holm need no dependence assumption at all. That is a genuine advantage, and it is often the reason to prefer them over BH when the correlation structure is unknown and the family is small.
5 · The arithmetic, spelled out
Write the sorted p-values as p(1) ≤ p(2) ≤ … ≤ p(m). Each method below is stated first as a rejection rule and then as the adjusted value that R’s p.adjust returns, because the two forms are equivalent and people meet both.
- Bonferroni. Reject every test with pi ≤ α/m. Adjusted value: min(1, m pi).
- Holm, step-down. Walk up the sorted list. Reject p(i) while p(i) ≤ α/(m − i + 1); stop at the first failure and retain everything from there on, whatever those p-values look like. Adjusted value: the running maximum of (m − i + 1) p(i), capped at 1. The running maximum is what keeps the column from decreasing.
- Benjamini–Hochberg, step-up. Find the largest k with p(k) ≤ kα/m, then reject ranks 1 through k. Note the direction: a test can be rejected even though its own p-value sits above its own line, because a larger p-value further down cleared its line. Adjusted value: the running minimum, taken from the largest p-value downward, of (m/i) p(i), capped at 1.
- Benjamini–Yekutieli. Exactly BH with every value multiplied by the harmonic sum 1 + ½ + … + 1/m.
Ties. Equal p-values must come out with equal adjusted values, or the ordering of a spreadsheet would decide which test was significant. The running maximum and running minimum handle this without a special case: two identical p-values are assigned the same running value, whichever order they were read in. This lab sorts with the input position as the tie-break, so repeating the same input always produces the same table.
Rounding. Nothing is rounded before the comparison to α. The table displays four significant figures; the CSV carries ten, and the decisions are made on the full-precision values. A p-value that reads 0.0500 on screen may be 0.05004 underneath, and it is the latter that decides.
6 · A worked example, checked by hand
An operations team tests ten candidate levers for an effect on on-time delivery — one test each, one p-value each. This is the example loaded into the box above. Take α = 0.05 and m = 10.
| i | p(i) | Bonferroni 10 p |
Holm (11 − i) p |
BH (10/i) p |
Holm line α/(11−i) |
BH line iα/10 |
|---|---|---|---|---|---|---|
| 1 | 0.0001 | 0.0010 | 0.0010 | 0.0010 | 0.00500 | 0.005 |
| 2 | 0.0042 | 0.0420 | 0.0378 | 0.0210 | 0.00556 | 0.010 |
| 3 | 0.0100 | 0.1000 | 0.0800 | 0.0333 | 0.00625 | 0.015 |
| 4 | 0.0180 | 0.1800 | 0.1260 | 0.0450 | 0.00714 | 0.020 |
| 5 | 0.0300 | 0.3000 | 0.1800 | 0.0600 | 0.00833 | 0.025 |
| 6 | 0.0440 | 0.4400 | 0.2200 | 0.0733 | 0.01000 | 0.030 |
| 7 | 0.0700 | 0.7000 | 0.2800 | 0.1000 | 0.01250 | 0.035 |
| 8 | 0.1500 | 1.0000 | 0.4500 | 0.1875 | 0.01667 | 0.040 |
| 9 | 0.3000 | 1.0000 | 0.6000 | 0.3333 | 0.02500 | 0.045 |
| 10 | 0.7000 | 1.0000 | 0.7000 | 0.7000 | 0.05000 | 0.050 |
- Unadjusted: six rejections. Six of the ten p-values are at or below 0.05. If all ten nulls were true and the tests were independent, the chance of at least one p-value below 0.05 is 1 − 0.9510 ≈ 0.40. Reporting six “significant” levers from this column would be reporting the multiplicity, not the levers.
- Bonferroni: two. The threshold is 0.05/10 = 0.005, and only 0.0001 and 0.0042 clear it. Equivalently, 10p is 0.001 and 0.042 for those two and 0.100 or more for everything else.
- Holm: two, but not by the same route. Rank 1: 0.0001 ≤ 0.05/10 = 0.00500, reject. Rank 2: 0.0042 ≤ 0.05/9 = 0.00556, reject. Rank 3: 0.0100 > 0.05/8 = 0.00625, stop. Everything from rank 3 down is retained without being tested. Holm matched Bonferroni here, which happens when the third-smallest p-value is far from the boundary; give rank 3 a p-value of 0.006 instead and Holm would reject three where Bonferroni still rejects two.
- Benjamini–Hochberg: four. Compare each p(i) with its own line i × 0.005. Rank 4 clears it (0.0180 ≤ 0.020); rank 5 does not (0.0300 > 0.025), nor does any rank below. The largest clearing rank is k = 4, so ranks 1 through 4 are rejected. Read down the BH adjusted column and the same four sit at or below 0.05: 0.0010, 0.0210, 0.0333, 0.0450.
- The step-up direction matters. Suppose rank 3 had been 0.0160 rather than 0.0100. It would then fail its own line (0.0160 > 0.015) — and BH would still reject it, because rank 4 clears at 0.0180 ≤ 0.020 and BH rejects everything up to the largest clearing rank. Holm cannot do this; it stops at the first failure and never looks further down.
- Benjamini–Yekutieli: one. The harmonic penalty at m = 10 is 1 + ½ + … + 1/10 = 7381/2520 ≈ 2.9290. Multiplying the BH column through gives 0.00293, 0.0615, 0.0977, 0.1318, … — only the first survives. That is the cost of dropping the dependence assumption, and it is why BY is worth reaching for deliberately rather than by default.
- What to report. Six, four, two or one, from the same ten numbers. The count is not a property of the data; it is a property of the promise you chose. Say which procedure you used, say why that promise fits the decision, and report the adjusted values rather than only the count.
References and further reading
- R Core Team, R: Adjust P-values for Multiple Comparisons (stats::p.adjust) — the definitions this lab reproduces, including the running-maximum and running-minimum formulations, the cap at 1 and the treatment of ties.
https://www.stat.ethz.ch/R-manual/R-devel/library/stats/html/p.adjust.html - Holm, S. (1979), “A simple sequentially rejective multiple test procedure”, Scandinavian Journal of Statistics 6(2), 65–70.
- Benjamini, Y. and Hochberg, Y. (1995), “Controlling the false discovery rate: a practical and powerful approach to multiple testing”, Journal of the Royal Statistical Society, Series B 57(1), 289–300.
- Benjamini, Y. and Yekutieli, D. (2001), “The control of the false discovery rate in multiple testing under dependency”, Annals of Statistics 29(4), 1165–1188 — the PRDS condition and the harmonic correction.
You can check adjusted values against R using p.adjust(p, method = "BH") or the corresponding method name. See References and acknowledgements for credits across the site.