What is a P-Value? Explained Simply

The Formal Definition of P-Value

The p-value is the probability of observing a test statistic at least as extreme as the one calculated from your sample data, assuming the null hypothesis is true. This definition is dense, so let us unpack it carefully. The null hypothesis (H₀) is your default assumption — for instance, that a drug has no effect, or that two groups have equal means.

When you calculate a p-value of 0.03, it means: if the null hypothesis were actually true, there would be only a 3% probability of getting results as extreme as yours by random chance alone. This is considered unlikely enough to cast doubt on the null hypothesis.

How P-Values Are Calculated

P-values come from comparing your test statistic to a theoretical probability distribution. Different tests use different distributions. A t-test uses the t-distribution, ANOVA uses the F-distribution, and chi-square tests use the chi-square distribution. Each distribution assigns probabilities to ranges of test statistic values.

For a two-tailed test, the p-value covers both extreme tails of the distribution — values more extreme in either direction. For a one-tailed test, only one tail is considered. Two-tailed tests are more conservative and are the default choice unless you have a strong directional hypothesis.

The Significance Threshold: Why 0.05?

The conventional threshold of α = 0.05 (5%) was introduced by Ronald Fisher in the 1920s as a rough guideline, not a sacred law. Fisher himself noted that researchers should use their own judgment. Yet the 0.05 threshold became entrenched in scientific publishing, creating an artificial boundary between "significant" and "not significant" results.

Many fields now use stricter thresholds: particle physics requires p < 0.000001 (5 sigma) before claiming a discovery; medical research often requires p < 0.01 for drug approval. The American Statistical Association has emphasised that p-values alone should not determine scientific conclusions.

What P-Values Cannot Tell You

P-values are widely misunderstood. Here is what a p-value does NOT tell you:

• It does not tell you the probability that the null hypothesis is true
• It does not tell you the probability that your results occurred by chance
• It does not measure the size or practical importance of an effect
• It does not tell you whether your study will replicate
• It does not account for whether your study was well designed

A common misinterpretation is "p = 0.04 means there is a 4% chance the null hypothesis is true." This is incorrect. The p-value is calculated assuming H₀ is true; it says nothing about the probability of H₀ itself.

P-Values and Sample Size: A Critical Relationship

One of the most important but underappreciated facts about p-values is their relationship with sample size. With a very large sample, even tiny, practically meaningless differences become statistically significant. With a small sample, even large, important differences may not reach significance.

For example, an online retailer with 1 million users might find that button colour A produces 0.001% more clicks than button colour B, with p < 0.001. The difference is statistically significant but economically trivial. Conversely, a clinical trial with 20 patients might fail to detect a genuine treatment effect simply because the study was underpowered.

Multiple Testing Problem

When you conduct multiple hypothesis tests simultaneously, the chance of getting at least one false positive increases dramatically. If you test 20 independent hypotheses each at α = 0.05, you would expect 1 false positive on average even if all null hypotheses are true. This is why large-scale studies (genetic association studies, brain imaging research) apply corrections like Bonferroni correction (divide α by number of tests) or false discovery rate (FDR) control.

Alternatives and Complements to P-Values

The scientific community increasingly recommends reporting confidence intervals alongside p-values. A 95% confidence interval shows the range of plausible values for the true parameter, conveying both statistical significance and practical meaningfulness. An effect size measure (Cohen's d, eta-squared, odds ratio) shows the magnitude of the effect independently of sample size.

Bayesian methods offer an alternative framework using Bayes factors, which directly compare the evidence for and against hypotheses. Unlike p-values, Bayes factors can provide evidence for the null hypothesis, which frequentist p-values cannot.

The Formal Definition of P-Value

The p-value is the probability of observing a test statistic at least as extreme as the one calculated from your sample data, assuming the null hypothesis is true. This definition is dense, so let us unpack it carefully. The null hypothesis (H₀) is your default assumption — for instance, that a drug has no effect, or that two groups have equal means.

When you calculate a p-value of 0.03, it means: if the null hypothesis were actually true, there would be only a 3% probability of getting results as extreme as yours by random chance alone. This is considered unlikely enough to cast doubt on the null hypothesis.

How P-Values Are Calculated

P-values come from comparing your test statistic to a theoretical probability distribution. Different tests use different distributions. A t-test uses the t-distribution, ANOVA uses the F-distribution, and chi-square tests use the chi-square distribution. Each distribution assigns probabilities to ranges of test statistic values.

For a two-tailed test, the p-value covers both extreme tails of the distribution — values more extreme in either direction. For a one-tailed test, only one tail is considered. Two-tailed tests are more conservative and are the default choice unless you have a strong directional hypothesis.

The Significance Threshold: Why 0.05?

The conventional threshold of α = 0.05 (5%) was introduced by Ronald Fisher in the 1920s as a rough guideline, not a sacred law. Fisher himself noted that researchers should use their own judgment. Yet the 0.05 threshold became entrenched in scientific publishing, creating an artificial boundary between "significant" and "not significant" results.

Many fields now use stricter thresholds: particle physics requires p < 0.000001 (5 sigma) before claiming a discovery; medical research often requires p < 0.01 for drug approval. The American Statistical Association has emphasised that p-values alone should not determine scientific conclusions.

What P-Values Cannot Tell You

P-values are widely misunderstood. Here is what a p-value does NOT tell you:

• It does not tell you the probability that the null hypothesis is true
• It does not tell you the probability that your results occurred by chance
• It does not measure the size or practical importance of an effect
• It does not tell you whether your study will replicate
• It does not account for whether your study was well designed

A common misinterpretation is "p = 0.04 means there is a 4% chance the null hypothesis is true." This is incorrect. The p-value is calculated assuming H₀ is true; it says nothing about the probability of H₀ itself.

P-Values and Sample Size: A Critical Relationship

One of the most important but underappreciated facts about p-values is their relationship with sample size. With a very large sample, even tiny, practically meaningless differences become statistically significant. With a small sample, even large, important differences may not reach significance.

For example, an online retailer with 1 million users might find that button colour A produces 0.001% more clicks than button colour B, with p < 0.001. The difference is statistically significant but economically trivial. Conversely, a clinical trial with 20 patients might fail to detect a genuine treatment effect simply because the study was underpowered.

Multiple Testing Problem

When you conduct multiple hypothesis tests simultaneously, the chance of getting at least one false positive increases dramatically. If you test 20 independent hypotheses each at α = 0.05, you would expect 1 false positive on average even if all null hypotheses are true. This is why large-scale studies (genetic association studies, brain imaging research) apply corrections like Bonferroni correction (divide α by number of tests) or false discovery rate (FDR) control.

Alternatives and Complements to P-Values

The scientific community increasingly recommends reporting confidence intervals alongside p-values. A 95% confidence interval shows the range of plausible values for the true parameter, conveying both statistical significance and practical meaningfulness. An effect size measure (Cohen's d, eta-squared, odds ratio) shows the magnitude of the effect independently of sample size.

Bayesian methods offer an alternative framework using Bayes factors, which directly compare the evidence for and against hypotheses. Unlike p-values, Bayes factors can provide evidence for the null hypothesis, which frequentist p-values cannot.

Complete Worked Example: Calculating a P-Value by Hand

A coin is suspected to be biased. You flip it 20 times and get 15 heads. What is the p-value for testing H₀: p = 0.5 (fair coin) vs H₁: p ≠ 0.5 (two-tailed)?

Under H₀, the number of heads X ~ Binomial(20, 0.5). The p-value is the probability of getting results as extreme as 15 or more extreme in either direction. P(X ≥ 15) = P(X=15) + P(X=16) + ... + P(X=20). By symmetry, P(X ≤ 5) = P(X ≥ 15). p-value = 2 × P(X ≥ 15) = 2 × [C(20,15)×0.5²⁰ + ... + C(20,20)×0.5²⁰] = 2 × (15504 + 4845 + 1140 + 190 + 20 + 1)/1048576 = 2 × 21700/1048576 ≈ 0.0414.

Since p = 0.041 < 0.05, reject H₀ at the 5% significance level. There is statistically significant evidence the coin is biased. However, note the 95% CI for p: [0.509, 0.908] — the bias could be modest or large, and the sample is small. The p-value says "significant"; the CI clarifies "we are uncertain how biased."

P-Value Interpretation: Five Common Misconceptions Corrected

Statistical education research consistently finds that most people — including researchers — misinterpret p-values. Here are the five most dangerous misconceptions with corrections:

Misconception 1: "p = 0.04 means there is a 4% probability the null hypothesis is true." Correction: The p-value is calculated assuming H₀ is true; it cannot give the probability that H₀ is true (that requires Bayesian analysis with a prior).

Misconception 2: "p = 0.06 means the result is almost significant." Correction: There is no continuum of "almost significant." Either the pre-specified threshold is crossed or it is not. "Trending toward significance" is not a statistical concept.

Misconception 3: "A small p-value means the effect is large." Correction: p-values conflate effect size with sample size. A tiny effect can produce p < 0.001 with a large sample. Always report effect sizes.

Misconception 4: "If p > 0.05, the null hypothesis is true." Correction: Failing to reject H₀ ≠ evidence for H₀. The study may simply be underpowered.

Misconception 5: "p = 0.049 and p = 0.051 are meaningfully different." Correction: The threshold is arbitrary. Both provide similar (weak) evidence against H₀. The evidence is nearly identical; the decision flip is an artefact of the threshold convention.