Resampling Two-sample inference Runs in your browser

Bootstrap Lab

Paste two independent samples. The lab estimates the difference between them, puts a percentile bootstrap interval around that difference, and tests it against a permutation null. Both procedures use a seed you control, so the same inputs give the same numbers every time you or a student runs them.

What this lab does, and what it does not

It does

Compute the observed difference B minus A for the statistic you pick (mean or median).
Draw seeded bootstrap resamples, resampling within each group separately, and report the 2.5th and 97.5th percentiles of the resampled differences as a 95% percentile interval.
Run a seeded two-sided permutation test by shuffling the pooled values, and report p = (1 + #{|t*| ≥ |t_obs|}) / (1 + R). Replicates that tie with the observed value count toward the numerator, which is why the comparison is ≥ and not >.
Do all of this in this page. Nothing is uploaded anywhere.

It does not

Correct the percentile interval for bias or skew. This is the plain percentile method, not BCa. On heavily skewed statistics the interval can sit off-centre; that is a property of the method, not a bug in the page.
Assume the two samples are paired. Everything here treats A and B as independent groups.
Handle missing values for you. Anything that is not a finite number is rejected with a message rather than silently dropped.

The two procedures answer different questions. The bootstrap interval asks how much the observed difference would wobble if you could redraw both samples from their own populations. The permutation test asks how often a difference this large shows up when the group labels carry no information at all. An interval that excludes zero and a small p-value usually travel together, but they are not the same calculation and they can disagree, especially with small or lopsided samples.

Set up the comparison

Data

Sample A

Numbers separated by commas, spaces, tabs, or new lines. At least two values.

Sample B

The statistic reported is B minus A, so put the group you think is higher here.

Settings

Statistic

Resamples (R)

Used for both the bootstrap and the permutation test.

Seed

Any text. Same seed and same data give identical output.

The interval level is fixed at 95%. Resampling runs in chunks of 200 so the page keeps responding while it works.

Results

Sample descriptions, the observed difference, the bootstrap interval, and the permutation test.
Quantity	Value	What it means

Bootstrap distribution.

Resampled differences Observed difference 95% interval bounds

Bootstrap histogram as a table

Every bar in the chart above, as bin edges and counts.
Bin	From	To	Count	Share

Permutation null distribution.

Differences under shuffled labels Observed difference, and its mirror image

Permutation histogram as a table

Every bar in the permutation chart, as bin edges and counts.
Bin	From	To	Count	Share

Export

Every export carries the seed, the resample count, and the statistic, so a downloaded file can be traced back to the run that produced it.

Self-tests

These checks run the same code the lab uses and compare it against values that can be worked out by hand or that have to hold by construction. Nothing here is pre-recorded: the numbers in the table are produced when you press the button. If a check fails, the arithmetic on this page is wrong and the results above should not be trusted.

Also callable from the console as window.runToolTests(), which returns the result object.

Method notes

The generator

Random numbers come from a 32-bit mulberry generator seeded by hashing your seed text. The bootstrap and the permutation test use two separate streams derived from that hash, so changing one procedure's resample count does not shift the other's draws. This is fine for teaching and for reproducibility. It is not a cryptographic generator and should not be used as one.

The percentile interval

Bootstrap replicates are sorted and the interval runs from the 2.5th to the 97.5th percentile, interpolating linearly between order statistics. With R resamples the interval is itself a random quantity; running the same data under a different seed will move the bounds slightly, and the smaller R is, the more it moves.

The permutation p-value

Pooled values are shuffled with a Fisher-Yates pass, split back into groups of the original sizes, and the statistic is recomputed. The plus-one in numerator and denominator counts the observed arrangement itself as one of the possible shuffles, which keeps the p-value from ever reaching zero. The smallest value the test can return is 1 / (1 + R): with R = 2000 that floor is 0.0005. A p-value sitting exactly at the floor means the test ran out of resolution, not that the difference is infinitely unlikely.

Ties

Replicates are counted when their absolute value is at least the observed absolute value, with a relative tolerance of 1e-12 to absorb floating-point noise. With small integer data, many shuffles reproduce the observed statistic exactly, and those all count. Dropping them would push the p-value down for no good reason.