Bayesian vs Frequentist Statistics

The Fundamental Philosophical Divide

Frequentist and Bayesian statistics represent two philosophically distinct approaches to probability and inference. Frequentists define probability as the long-run frequency of an event in repeated identical experiments. Bayesians define probability as a degree of belief that can be updated as evidence accumulates. This seemingly abstract distinction has profound practical consequences for how we conduct and interpret statistical analyses.

The Frequentist Approach

Frequentist statistics — built by Fisher, Neyman, and Pearson — treats parameters as fixed (though unknown) values and data as random. P-values, confidence intervals, and hypothesis tests are the main tools. A 95% confidence interval means that 95% of intervals constructed by this procedure would contain the true parameter — not that there is a 95% probability the parameter is in this specific interval. This framework dominates most published science.

The Bayesian Approach

Bayesian statistics treats parameters as random variables with probability distributions representing uncertainty. The prior distribution P(θ) encodes beliefs before seeing data. The likelihood P(data|θ) measures how probable the data is given each parameter value. Bayes' theorem combines these: P(θ|data) ∝ P(data|θ) × P(θ). The posterior distribution P(θ|data) is the updated belief after observing data.

Prior Distributions: Strength and Controversy

The prior distribution is the most distinctive and controversial element of Bayesian analysis. Informative priors encode genuine prior knowledge (from previous studies, expert opinion, or physical constraints). Weakly informative priors provide regularisation without strong assumptions. Non-informative (flat) priors attempt to let data dominate. The sensitivity of conclusions to the prior choice is always worth examining — if results change substantially with different reasonable priors, conclusions depend heavily on prior beliefs.

Practical Differences in Interpretation

Bayesian credible intervals are directly interpretable: "There is a 95% probability the parameter lies in [a, b]." This is what many people wrongly think frequentist confidence intervals mean. Bayesian hypothesis testing uses Bayes factors — ratios of marginal likelihoods — rather than p-values. A Bayes factor of 10 means the data is 10 times more probable under H₁ than H₀. Unlike p-values, Bayes factors can provide evidence for H₀.

When Each Approach Excels

Bayesian methods excel for: sequential analysis (updating beliefs as data accumulates), small samples where prior knowledge is valuable, complex hierarchical models, predictions rather than hypothesis testing, and when direct probability statements about parameters are needed. Frequentist methods are preferred for: regulatory contexts with established standards (drug approval, clinical trials), situations where prior specification is contested, and simple analyses where both methods give similar results.

The Modern Landscape: Pragmatic Synthesis

The historical debate has softened considerably. Modern statisticians increasingly use both frameworks pragmatically, choosing based on the problem rather than ideology. Multilevel models are often implemented Bayesianly (using MCMC sampling) while reporting frequentist-style results. Many Bayesian analyses with non-informative priors give numerically similar results to frequentist analyses, while providing more interpretable output.

The Fundamental Philosophical Divide

Frequentist and Bayesian statistics represent two philosophically distinct approaches to probability and inference. Frequentists define probability as the long-run frequency of an event in repeated identical experiments. Bayesians define probability as a degree of belief that can be updated as evidence accumulates. This seemingly abstract distinction has profound practical consequences for how we conduct and interpret statistical analyses.

The Frequentist Approach

Frequentist statistics — built by Fisher, Neyman, and Pearson — treats parameters as fixed (though unknown) values and data as random. P-values, confidence intervals, and hypothesis tests are the main tools. A 95% confidence interval means that 95% of intervals constructed by this procedure would contain the true parameter — not that there is a 95% probability the parameter is in this specific interval. This framework dominates most published science.

The Bayesian Approach

Bayesian statistics treats parameters as random variables with probability distributions representing uncertainty. The prior distribution P(θ) encodes beliefs before seeing data. The likelihood P(data|θ) measures how probable the data is given each parameter value. Bayes' theorem combines these: P(θ|data) ∝ P(data|θ) × P(θ). The posterior distribution P(θ|data) is the updated belief after observing data.

Prior Distributions: Strength and Controversy

The prior distribution is the most distinctive and controversial element of Bayesian analysis. Informative priors encode genuine prior knowledge (from previous studies, expert opinion, or physical constraints). Weakly informative priors provide regularisation without strong assumptions. Non-informative (flat) priors attempt to let data dominate. The sensitivity of conclusions to the prior choice is always worth examining — if results change substantially with different reasonable priors, conclusions depend heavily on prior beliefs.

Practical Differences in Interpretation

Bayesian credible intervals are directly interpretable: "There is a 95% probability the parameter lies in [a, b]." This is what many people wrongly think frequentist confidence intervals mean. Bayesian hypothesis testing uses Bayes factors — ratios of marginal likelihoods — rather than p-values. A Bayes factor of 10 means the data is 10 times more probable under H₁ than H₀. Unlike p-values, Bayes factors can provide evidence for H₀.

When Each Approach Excels

Bayesian methods excel for: sequential analysis (updating beliefs as data accumulates), small samples where prior knowledge is valuable, complex hierarchical models, predictions rather than hypothesis testing, and when direct probability statements about parameters are needed. Frequentist methods are preferred for: regulatory contexts with established standards (drug approval, clinical trials), situations where prior specification is contested, and simple analyses where both methods give similar results.

The Modern Landscape: Pragmatic Synthesis

The historical debate has softened considerably. Modern statisticians increasingly use both frameworks pragmatically, choosing based on the problem rather than ideology. Multilevel models are often implemented Bayesianly (using MCMC sampling) while reporting frequentist-style results. Many Bayesian analyses with non-informative priors give numerically similar results to frequentist analyses, while providing more interpretable output.

MCMC and Computational Bayesian Methods

One historical barrier to Bayesian methods was computational: analytically deriving posterior distributions is only possible for specific prior-likelihood combinations (conjugate pairs). Markov Chain Monte Carlo (MCMC) methods — particularly Gibbs sampling and the Metropolis-Hastings algorithm — revolutionised Bayesian computation by drawing samples from posterior distributions numerically. Modern tools like Stan and PyMC allow Bayesian modelling of virtually any data structure. Hamiltonian Monte Carlo (HMC), used in Stan, is dramatically more efficient than older MCMC methods, making complex hierarchical models tractable.

Bayesian vs Frequentist: A Worked Comparison

Same dataset, two frameworks. A factory claims their defect rate is 2%. You inspect 200 items and find 8 defects (4%). Is the claim credible?

Frequentist approach: H₀: p = 0.02. Test statistic: z = (0.04−0.02)/√(0.02×0.98/200) = 0.02/0.00990 = 2.02. p-value = 2×P(Z > 2.02) = 0.043. At α = 0.05, reject H₀. Conclusion: statistically significant evidence against the 2% claim.

Bayesian approach: Prior: Beta(α=2, β=98) — encoding belief that defect rate is around 2%, with moderate uncertainty. Likelihood: 8 defects in 200 trials. Posterior: Beta(2+8, 98+192) = Beta(10, 290). Posterior mean = 10/300 = 3.33%. 95% credible interval: [1.61%, 6.00%]. The factory's claimed 2% lies within the credible interval — the Bayesian approach, incorporating the factory's prior reputation, gives a more nuanced answer. The frequentist test rejected the claim; the Bayesian analysis says the data is consistent with 2% being plausible but the point estimate shifted toward 3.3%.

The Choice in Practice: A Decision Guide

Choosing between frameworks should be practical, not dogmatic. Use frequentist methods when: you have no meaningful prior information, your audience expects standard p-values and CIs (regulators, journal editors), you need a simple defensible analysis, or computational resources are limited. Use Bayesian methods when: you have genuine, justified prior knowledge to incorporate (previous trials, physical constraints), you need to update analyses sequentially as data accumulates, you want direct probability statements about parameters, or you are fitting complex hierarchical models where Bayesian MCMC is practically essential. In modern practice, the question is not "which is right?" but "which is most appropriate for this problem?"