Confidence Intervals Explained

What a Confidence Interval Actually Means

A 95% confidence interval is frequently misinterpreted. It does NOT mean "there is a 95% probability the true parameter lies in this interval." The parameter is fixed (though unknown) — it is either in the interval or it is not. The correct interpretation is: if we repeated the study many times and constructed a confidence interval each time, 95% of those intervals would contain the true parameter.

This frequentist interpretation is subtle but important. Once you have computed a specific interval like [2.1, 5.8], that specific interval either contains the truth or it does not. The 95% refers to the procedure's long-run reliability, not to the probability for any single interval.

How Confidence Intervals Are Constructed

For a population mean with known σ, the CI is: x̄ ± z*(σ/√n), where z* is the critical value from the standard normal distribution. For α = 0.05 (95% CI), z* = 1.96. For α = 0.01 (99% CI), z* = 2.576.

When σ is unknown (the common practical case), replace σ with s and z* with t* from the t-distribution with n−1 degrees of freedom. As n increases, t* approaches z*, which is why large samples give similar results whether you use z or t.

Width of Confidence Intervals

The width of a CI is determined by three factors: confidence level (higher confidence → wider interval), sample size (larger n → narrower interval), and variability (higher SD → wider interval). There is always a tradeoff between confidence and precision. A 99% CI is wider than a 95% CI for the same data — you gain certainty at the cost of precision.

Margin of error = z* × (s/√n) = half the CI width. Surveys and polls report margins of error, which correspond to the half-width of a 95% confidence interval.

Confidence Intervals vs Hypothesis Tests

Confidence intervals and hypothesis tests are mathematically equivalent and carry the same information. If a 95% CI for the difference between two means excludes zero, the two-tailed hypothesis test at α = 0.05 would reject the null hypothesis of equal means. If the CI includes zero, the test would fail to reject.

However, CIs are generally preferred in scientific reporting because they convey both statistical significance AND practical magnitude. A hypothesis test only answers "is there an effect?" while a CI answers "how big is the effect, and how precisely do we know it?"

Confidence Intervals for Proportions

For proportions, the CI formula changes. The Wald interval (most common) is: p̂ ± z* × √(p̂(1−p̂)/n). For example, a poll finding 52% support with n=1000: CI = 0.52 ± 1.96×√(0.52×0.48/1000) = 0.52 ± 0.031 = [0.489, 0.551].

The Wald interval performs poorly for proportions near 0 or 1. The Wilson score interval is recommended instead, especially for extreme proportions and small samples.

Bootstrap Confidence Intervals

When theoretical distributional assumptions are questionable, bootstrap confidence intervals provide a flexible, assumption-free alternative. The bootstrap involves: resampling your data with replacement thousands of times, computing the statistic of interest for each resample, and using the distribution of those bootstrap statistics to construct the CI.

This approach works for almost any statistic (median, correlation, regression coefficient) and makes no distributional assumptions. It is computationally intensive but trivial with modern computers.

What a Confidence Interval Actually Means

A 95% confidence interval is frequently misinterpreted. It does NOT mean "there is a 95% probability the true parameter lies in this interval." The parameter is fixed (though unknown) — it is either in the interval or it is not. The correct interpretation is: if we repeated the study many times and constructed a confidence interval each time, 95% of those intervals would contain the true parameter.

This frequentist interpretation is subtle but important. Once you have computed a specific interval like [2.1, 5.8], that specific interval either contains the truth or it does not. The 95% refers to the procedure's long-run reliability, not to the probability for any single interval.

How Confidence Intervals Are Constructed

For a population mean with known σ, the CI is: x̄ ± z*(σ/√n), where z* is the critical value from the standard normal distribution. For α = 0.05 (95% CI), z* = 1.96. For α = 0.01 (99% CI), z* = 2.576.

When σ is unknown (the common practical case), replace σ with s and z* with t* from the t-distribution with n−1 degrees of freedom. As n increases, t* approaches z*, which is why large samples give similar results whether you use z or t.

Width of Confidence Intervals

The width of a CI is determined by three factors: confidence level (higher confidence → wider interval), sample size (larger n → narrower interval), and variability (higher SD → wider interval). There is always a tradeoff between confidence and precision. A 99% CI is wider than a 95% CI for the same data — you gain certainty at the cost of precision.

Margin of error = z* × (s/√n) = half the CI width. Surveys and polls report margins of error, which correspond to the half-width of a 95% confidence interval.

Confidence Intervals vs Hypothesis Tests

Confidence intervals and hypothesis tests are mathematically equivalent and carry the same information. If a 95% CI for the difference between two means excludes zero, the two-tailed hypothesis test at α = 0.05 would reject the null hypothesis of equal means. If the CI includes zero, the test would fail to reject.

However, CIs are generally preferred in scientific reporting because they convey both statistical significance AND practical magnitude. A hypothesis test only answers "is there an effect?" while a CI answers "how big is the effect, and how precisely do we know it?"

Confidence Intervals for Proportions

For proportions, the CI formula changes. The Wald interval (most common) is: p̂ ± z* × √(p̂(1−p̂)/n). For example, a poll finding 52% support with n=1000: CI = 0.52 ± 1.96×√(0.52×0.48/1000) = 0.52 ± 0.031 = [0.489, 0.551].

The Wald interval performs poorly for proportions near 0 or 1. The Wilson score interval is recommended instead, especially for extreme proportions and small samples.

Bootstrap Confidence Intervals

When theoretical distributional assumptions are questionable, bootstrap confidence intervals provide a flexible, assumption-free alternative. The bootstrap involves: resampling your data with replacement thousands of times, computing the statistic of interest for each resample, and using the distribution of those bootstrap statistics to construct the CI.

This approach works for almost any statistic (median, correlation, regression coefficient) and makes no distributional assumptions. It is computationally intensive but trivial with modern computers.

Worked Example: Constructing a Confidence Interval from Scratch

A researcher measures the daily screen time (hours) of a random sample of 36 university students: x̄ = 6.4 hours, s = 2.1 hours. She wants a 99% confidence interval for the population mean.

Since σ is unknown, use the t-distribution with df = 35. For 99% CI (α/2 = 0.005), t* = 2.724 (from t-table or calculator). Margin of error = t* × s/√n = 2.724 × 2.1/√36 = 2.724 × 0.35 = 0.953. CI: [6.4 − 0.953, 6.4 + 0.953] = [5.45, 7.35] hours.

Interpretation: We are 99% confident that the true mean daily screen time for all university students lies between 5.45 and 7.35 hours. If we repeated this study 100 times with different random samples and built a 99% CI each time, 99 of those intervals would contain the true population mean.

Notice the 99% CI [5.45, 7.35] is wider than what a 95% CI [5.69, 7.11] would be — higher confidence requires a wider net. The sample mean of 6.4 hours is always in the centre of the interval; uncertainty is symmetric in both directions.

Visualising Confidence Intervals: Forest Plots

In meta-analysis and systematic reviews, forest plots display confidence intervals from multiple studies simultaneously. Each study is represented as a horizontal line (the CI) with a square (the point estimate, sized proportional to the study's weight). A diamond at the bottom shows the pooled estimate and its CI. If the pooled CI excludes zero (or the null value), the meta-analysis is statistically significant. Forest plots make it easy to spot heterogeneity (inconsistency across studies) — if CIs barely overlap or point in different directions, the studies may be measuring different things or have important methodological differences.

Confidence Intervals for Difference Between Two Means

Two teaching methods are compared: Method A (n=40): x̄₁=78, s₁=12. Method B (n=35): x̄₂=73, s₂=14. 95% CI for the difference μ₁−μ₂ using Welch's formula: SE_diff = √(12²/40 + 14²/35) = √(3.6 + 5.6) = √9.2 = 3.03. df ≈ 68 (Welch-Satterthwaite). t*(68) ≈ 2.00. CI: (78−73) ± 2.00×3.03 = 5 ± 6.06 = [−1.06, 11.06]. The CI includes zero — no statistically significant difference at 95% confidence. Despite a 5-point observed difference, it could plausibly be due to sampling variation.