P-Values and Hypothesis Testing Explained With a Coin-Flip Example

Calculating the p-Value

The test statistic is z = (60 − 50) ÷ 5 = 2.0. The probability of getting a z of 2.0 or higher is about 2.3% in one direction. Because a coin could be biased toward heads or tails, we use a two-sided test and double it: 4.6%. Using a continuity correction (treating 60 as 59.5), the two-sided p-value is about 5.7%. The exact binomial calculation gives roughly 5.7% as well.

At the conventional 5% significance level, the answer is borderline. Under the plain z-test, p = 0.0455 would be called significant, while the exact test, with p = 0.057, would not. This shows how sensitive conclusions near the threshold can be.

What the p-Value Means

The p-value is the probability of seeing data at least this extreme if the null hypothesis is true. It is not the probability that the null hypothesis is true, and it is not the probability that your result is due to chance. A p-value of 0.03 does not mean there is a 97% chance the effect is real. To make that kind of statement, you would need prior probabilities, as in Bayesian analysis.

A threshold like 0.05 is a convention, not a law of nature. Results with p = 0.049 and p = 0.051 are practically equivalent evidence.

Significance Level and Errors

The significance level (alpha), often 0.05, is the false-positive rate you accept: the chance of rejecting a true null hypothesis (a Type I error). The opposite error, failing to reject a false null, is a Type II error, and its probability depends on sample size, effect size, and alpha. The power of a test is 1 minus the Type II error rate. Larger samples increase power. A study that finds no significant effect with low power does not prove there is no effect.

Running many tests increases the chance of at least one false positive: with 20 independent tests at alpha = 0.05, the probability of at least one false positive is 1 − 0.95²⁰ = 64%. Corrections such as Bonferroni adjust for this.

Statistical vs. Practical Significance

With a large enough sample, even tiny differences can produce small p-values. Suppose an ad campaign increases a conversion rate from 2.00% to 2.05% in a test of millions of users. This may be statistically significant but meaningless in business terms. Always look at the size of the effect and its confidence interval, not only the p-value.

Common Misinterpretations

Treating p > 0.05 as proof of no effect; reporting only significant results; stopping an experiment as soon as p dips below 0.05 (peeking); assuming a small p-value means a large or important effect; and ignoring assumptions such as independence and random sampling. Pre-register your analysis plan where possible and report effect sizes.