What is Standard Deviation?

Population vs Sample Standard Deviation

One of the most important distinctions in statistics is between population and sample standard deviation. When you have data from an entire population, you use the population formula dividing by n. When you have a sample taken from a larger population, you divide by n−1 (called Bessel's correction). This correction prevents underestimation of variability when working with samples.

In practice, the difference becomes negligible for large samples (n > 30), but for small samples the correction matters considerably. Most statistical software defaults to sample standard deviation for this reason.

Standard Deviation vs Variance

Variance (s²) and standard deviation (s) are closely related — SD is simply the square root of variance. Variance has useful mathematical properties (it is additive for independent variables), but its units are squared, making interpretation difficult. Standard deviation restores the original units, making it more intuitive for communication.

For example, if measuring heights in centimetres, variance is in cm², which is hard to visualise. Standard deviation is back in cm, easily comparable to the mean height.

Interpreting Standard Deviation with the Empirical Rule

For data that follows a normal distribution, the empirical rule (also called the 68-95-99.7 rule) provides an extremely useful interpretation framework:

• 68% of data falls within 1 standard deviation of the mean (μ ± σ)
• 95% of data falls within 2 standard deviations (μ ± 2σ)
• 99.7% of data falls within 3 standard deviations (μ ± 3σ)

This means only 0.3% of normally distributed data lies beyond 3 standard deviations. In quality control, a process operating within 3σ limits is considered in control. Six Sigma manufacturing aims to reduce defects so that the process mean is 6 standard deviations from the nearest specification limit.

Standard Deviation in Finance and Risk Analysis

In finance, standard deviation of investment returns is the most widely used measure of risk. A stock with monthly returns having an SD of 8% is considerably more volatile than one with SD of 2%. Portfolio managers use this metric to construct diversified portfolios that achieve the best risk-return tradeoff.

The Sharpe ratio — a cornerstone of modern portfolio theory — divides excess return by standard deviation: Sharpe = (Return − Risk-free rate) / SD. A higher Sharpe ratio means better risk-adjusted performance.

Common Mistakes When Using Standard Deviation

Many analysts make errors when applying standard deviation. The most common mistake is using SD to describe skewed data — standard deviation assumes the data is roughly symmetric. For heavily skewed distributions (like income data or house prices), the interquartile range (IQR) or median absolute deviation (MAD) are better spread measures.

Another mistake is confusing standard deviation with standard error. Standard deviation describes spread of individual data points. Standard error describes how much the sample mean itself varies from sample to sample, and equals SD / √n.

Calculating Standard Deviation by Hand: Full Worked Example

Let's walk through a complete calculation with a realistic dataset. A teacher records quiz scores for 6 students: 72, 85, 90, 68, 95, 88.

Step 1 — Calculate the mean: x̄ = (72+85+90+68+95+88)/6 = 498/6 = 83.0

Step 2 — Compute deviations from mean:
72−83 = −11, squared = 121
85−83 = 2, squared = 4
90−83 = 7, squared = 49
68−83 = −15, squared = 225
95−83 = 12, squared = 144
88−83 = 5, squared = 25

Step 3 — Sum of squared deviations: SS = 121+4+49+225+144+25 = 568

Step 4 — Sample variance: s² = 568/(6−1) = 568/5 = 113.6

Step 5 — Standard deviation: s = √113.6 ≈ 10.66

Interpretation: the typical student score deviates from the class average of 83 by about 10.66 points in either direction.

Standard Deviation in Machine Learning

Standard deviation plays a central role in data preprocessing for machine learning. Feature scaling (standardisation) transforms each feature by subtracting its mean and dividing by its SD, producing z-scores. This ensures all features contribute equally to algorithms sensitive to magnitude, such as k-nearest neighbours, support vector machines, and neural networks.

In model evaluation, SD of cross-validation scores tells you how consistent the model is across different data splits. High SD in CV scores suggests the model is sensitive to which data it is trained on — a sign of overfitting or insufficient data.

Population vs Sample Standard Deviation

One of the most important distinctions in statistics is between population and sample standard deviation. When you have data from an entire population, you use the population formula dividing by n. When you have a sample taken from a larger population, you divide by n−1 (called Bessel's correction). This correction prevents underestimation of variability when working with samples.

In practice, the difference becomes negligible for large samples (n > 30), but for small samples the correction matters considerably. Most statistical software defaults to sample standard deviation for this reason.

Standard Deviation vs Variance

Variance (s²) and standard deviation (s) are closely related — SD is simply the square root of variance. Variance has useful mathematical properties (it is additive for independent variables), but its units are squared, making interpretation difficult. Standard deviation restores the original units, making it more intuitive for communication.

For example, if measuring heights in centimetres, variance is in cm², which is hard to visualise. Standard deviation is back in cm, easily comparable to the mean height.

Interpreting Standard Deviation with the Empirical Rule

For data that follows a normal distribution, the empirical rule (also called the 68-95-99.7 rule) provides an extremely useful interpretation framework:

• 68% of data falls within 1 standard deviation of the mean (μ ± σ)
• 95% of data falls within 2 standard deviations (μ ± 2σ)
• 99.7% of data falls within 3 standard deviations (μ ± 3σ)

This means only 0.3% of normally distributed data lies beyond 3 standard deviations. In quality control, a process operating within 3σ limits is considered in control. Six Sigma manufacturing aims to reduce defects so that the process mean is 6 standard deviations from the nearest specification limit.

Standard Deviation in Finance and Risk Analysis

In finance, standard deviation of investment returns is the most widely used measure of risk. A stock with monthly returns having an SD of 8% is considerably more volatile than one with SD of 2%. Portfolio managers use this metric to construct diversified portfolios that achieve the best risk-return tradeoff.

The Sharpe ratio — a cornerstone of modern portfolio theory — divides excess return by standard deviation: Sharpe = (Return − Risk-free rate) / SD. A higher Sharpe ratio means better risk-adjusted performance.

Common Mistakes When Using Standard Deviation

Many analysts make errors when applying standard deviation. The most common mistake is using SD to describe skewed data — standard deviation assumes the data is roughly symmetric. For heavily skewed distributions (like income data or house prices), the interquartile range (IQR) or median absolute deviation (MAD) are better spread measures.

Another mistake is confusing standard deviation with standard error. Standard deviation describes spread of individual data points. Standard error describes how much the sample mean itself varies from sample to sample, and equals SD / √n.

Calculating Standard Deviation by Hand: Full Worked Example

Let's walk through a complete calculation with a realistic dataset. A teacher records quiz scores for 6 students: 72, 85, 90, 68, 95, 88.

Step 1 — Calculate the mean: x̄ = (72+85+90+68+95+88)/6 = 498/6 = 83.0

Step 2 — Compute deviations from mean:
72−83 = −11, squared = 121
85−83 = 2, squared = 4
90−83 = 7, squared = 49
68−83 = −15, squared = 225
95−83 = 12, squared = 144
88−83 = 5, squared = 25

Step 3 — Sum of squared deviations: SS = 121+4+49+225+144+25 = 568

Step 4 — Sample variance: s² = 568/(6−1) = 568/5 = 113.6

Step 5 — Standard deviation: s = √113.6 ≈ 10.66

Interpretation: the typical student score deviates from the class average of 83 by about 10.66 points in either direction.

Standard Deviation in Machine Learning

Standard deviation plays a central role in data preprocessing for machine learning. Feature scaling (standardisation) transforms each feature by subtracting its mean and dividing by its SD, producing z-scores. This ensures all features contribute equally to algorithms sensitive to magnitude, such as k-nearest neighbours, support vector machines, and neural networks.

In model evaluation, SD of cross-validation scores tells you how consistent the model is across different data splits. High SD in CV scores suggests the model is sensitive to which data it is trained on — a sign of overfitting or insufficient data.