Machine Learning · Fundamentals
Statistic · estimator · bias · variance · mean squared error
Every number we report about a model — a test error, an accuracy, a fitted parameter — is computed from a finite, random sample of data. A different sample would have given a different number.
This recap supplies the vocabulary for that — estimator, bias, variance — which the bias–variance trade-off lifts from a number to a whole model. How large the ± of a single estimate is follows in the recap Uncertainty.
Prerequisite: the recap Probability — random variables, expectation, variance, the binomial.
| Part | The question it answers |
|---|---|
| 1 · Statistic and estimator | What is an estimate, and why is it random? |
| 2 · Bias and variance | Is an estimator right on average, and how much does it scatter? |
| 3 · The mean squared error | How do the two combine into one error? |
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
np.set_printoptions(precision=4, suppress=True)
plt.rcParams["figure.figsize"] = (9, 4)
plt.rcParams["axes.grid"] = True
plt.rcParams["grid.alpha"] = 0.3
rng = np.random.default_rng(0)
A test error, an accuracy, a fitted parameter: each is a number computed from random data. Before we can say how good such a number is, we need one word for it — an estimator — and two words for its quality: bias and variance.
The bias–variance trade-off (→ L-BiasVar) applies exactly the same algebra to whole models.
$$ \mathcal{D} = \{X_1, X_2, \dots, X_n\} $$
Here we regard the data as random variables (the frequentist view), and assume they are i.i.d.:
A statistic is a random variable $S$ that is a function of the data:
$$ S = f(\mathcal{D}) $$
An estimator is a statistic intended to approximate a parameter governing the distribution of $\mathcal{D}$.
Notation: $\hat\theta$ is an estimator of $\theta$.
An estimator is a random variable: it has a distribution, because $\mathcal{D}$ is random. Different data sets give different values of $\hat\theta$.
That is what makes "bias" and "variance" meaningful.
$$ \operatorname{bias}(\hat\theta) := \mathbb{E}\left[\hat\theta\right] - \theta $$
The difference between the mean of the estimator and the true value of $\theta$.
The expectation is taken over different data sets $\mathcal{D}$.
An estimator is unbiased if
$$ \operatorname{bias}(\hat\theta) = 0 \qquad\Longleftrightarrow\qquad \mathbb{E}\left[\hat\theta\right] = \theta . $$
$$ \operatorname{var}(\hat\theta) = \mathbb{E}\left[\left(\hat\theta - \mathbb{E}[\hat\theta]\right)^2\right] = \mathbb{E}[\hat\theta^2] - (\mathbb{E}[\hat\theta])^2 $$
The shortcut uses the rules of variance (Probability): $\mathbb E[\hat\theta]$ is a number, so expanding the square and using linearity gives $\mathbb E[\hat\theta^2] - 2(\mathbb E\hat\theta)^2 + (\mathbb E\hat\theta)^2$.
The expectation is again over data sets $\mathcal D$.
Data generating distribution:
$$ X_i \sim \mathcal{N}(\mu, \sigma^2) $$
Two parameters to estimate:
Careful with the word "variance". It is used here for two different things:
- the variance of the estimator, $\operatorname{var}(\hat\theta)$
- the variance of the Gaussian, $\sigma^2$ — which is itself the quantity being estimated
Sample mean, an estimator of $\mu$:
$$ \hat\mu = \frac{1}{n}\sum_{i=1}^{n} X_i =: \bar X $$
(Biased) sample variance:
$$ \hat\sigma_b^2 = \frac{1}{n}\sum_{i=1}^{n}(X_i - \bar X)^2 $$
(Unbiased) sample variance:
$$ \hat\sigma_u^2 = \frac{1}{n-1}\sum_{i=1}^{n}(X_i - \bar X)^2 $$
$$ \mathbb{E}\left[\hat\mu\right] = \mathbb{E}\left[\frac{1}{n}\sum_i X_i\right] = \frac{1}{n}\sum_i \mathbb{E}[X_i] = \frac{1}{n}\sum_i \mu = \mu $$
so $\operatorname{bias}(\hat\mu) = 0$.
The rules of variance (Probability) — a constant factor comes out squared, and for independent $X_i$ the variances add:
$$ \operatorname{var}(\hat\mu) = \operatorname{var}\Bigl(\frac1n \sum_{i=1}^n X_i\Bigr) = \frac{1}{n^2}\operatorname{var}\Bigl(\sum_{i=1}^n X_i\Bigr) = \frac{1}{n^2}\sum_{i=1}^n \operatorname{var}(X_i) = \frac{1}{n^2}\, n\sigma^2 = \frac{\sigma^2}{n} $$
$\operatorname{var}(\hat\mu) = \sigma^2/n$ — the estimator gets more reliable with more data, and its spread shrinks like $1/\sqrt{n}$.
$\hat\sigma_b^2$, in contrast, is biased: $\mathbb{E}[\hat\sigma_b^2] = \frac{n-1}{n}\sigma^2$ (next two slides), i.e. $\operatorname{bias}(\hat\sigma_b^2) = -\sigma^2/n$. It systematically underestimates, because $\bar X$ is fitted to the same data.
Write each deviation from the fitted mean as $X_i - \bar X = (X_i - \mu) - (\bar X - \mu)$, expand the square, and use $\sum_i (X_i - \mu) = n(\bar X - \mu)$:
$$ \sum_{i}(X_i - \bar X)^2 = \sum_i (X_i - \mu)^2 - 2(\bar X - \mu)\sum_i (X_i - \mu) + n(\bar X - \mu)^2 = \sum_i (X_i - \mu)^2 - n(\bar X - \mu)^2 $$
The deviations from $\bar X$ are smaller than those from $\mu$ — by exactly $n(\bar X - \mu)^2 \ge 0$.
Take expectations, with $\mathbb E[(X_i - \mu)^2] = \sigma^2$ and $\mathbb E[(\bar X - \mu)^2] = \operatorname{var}(\bar X) = \sigma^2/n$:
$$ \mathbb E\Bigl[\sum_i (X_i - \bar X)^2\Bigr] = n\sigma^2 - \sigma^2 = (n-1)\,\sigma^2 \quad\Longrightarrow\quad \mathbb E[\hat\sigma_b^2] = \frac{n-1}{n}\sigma^2, \qquad \mathbb E[\hat\sigma_u^2] = \sigma^2 $$
Dividing by $n - 1$ instead of $n$ (Bessel's correction) removes the bias.
Draw many data sets, and look at the distribution of each estimator.
n, mu_true, sigma_true, reps = 5, 3.0, 2.0, 100_000
S = rng.normal(mu_true, sigma_true, size=(reps, n)) # one row = one data set D
mu_hat = S.mean(axis=1)
var_b = ((S - mu_hat[:, None]) ** 2).mean(axis=1) # 1/n
var_u = ((S - mu_hat[:, None]) ** 2).sum(axis=1) / (n - 1) # 1/(n-1)
print(f"{reps} data sets of n = {n}")
print(f"E[mu_hat] = {mu_hat.mean():.4f} bias = {mu_hat.mean() - mu_true:+.4f} (theory 0)")
print(f"var(mu_hat) = {mu_hat.var():.4f} theory sigma^2/n = {sigma_true**2 / n:.4f}")
print(f"E[sigma_b^2] = {var_b.mean():.4f} bias = {var_b.mean() - sigma_true**2:+.4f}"
f" (theory {-sigma_true**2 / n:+.4f})")
print(f"E[sigma_u^2] = {var_u.mean():.4f} bias = {var_u.mean() - sigma_true**2:+.4f} (theory 0)")
fig, ax = plt.subplots(1, 2, figsize=(11, 3.6))
ax[0].hist(mu_hat, bins=60, color="steelblue")
ax[0].axvline(mu_true, color="k", lw=2, label=r"true $\mu$")
ax[0].set_title(r"distribution of $\hat\mu$"); ax[0].legend()
ax[1].hist(var_b, bins=60, range=(0, 20), color="indianred", alpha=.7, label=r"$\hat\sigma_b^2$")
ax[1].hist(var_u, bins=60, range=(0, 20), color="seagreen", alpha=.5, label=r"$\hat\sigma_u^2$")
ax[1].axvline(sigma_true**2, color="k", lw=2, label=r"true $\sigma^2$")
ax[1].set_title("distribution of the variance estimators"); ax[1].legend()
plt.tight_layout(); plt.show()
$$ \operatorname{mse}(\hat\theta) = \mathbb{E}\left[\left(\theta - \hat\theta\right)^2\right] $$
This is what we actually care about: how far the estimate lands from the truth, on average.
$\theta$ is a constant, so linearity (Probability) gives the second line; then add and subtract $(\mathbb E[\hat\theta])^2$:
$$ \begin{aligned} \operatorname{mse}(\hat\theta) &= \mathbb{E}\left[(\theta - \hat\theta)^2\right] \\ &= \theta^2 - 2\theta\,\mathbb{E}[\hat\theta] + \mathbb{E}[\hat\theta^2] \;\underbrace{+\; (\mathbb{E}[\hat\theta])^2 - (\mathbb{E}[\hat\theta])^2}_{= \,0} \end{aligned} $$
Regrouping the six terms into two brackets:
$$ \begin{aligned} &= \underbrace{\left(\theta^2 - 2\theta\,\mathbb{E}[\hat\theta] + (\mathbb{E}[\hat\theta])^2\right)}_{(\mathbb{E}[\hat\theta] - \theta)^2} + \underbrace{\left(\mathbb{E}[\hat\theta^2] - (\mathbb{E}[\hat\theta])^2\right)}_{\operatorname{var}(\hat\theta)} \end{aligned} $$
$$ \boxed{\;\operatorname{mse}(\hat\theta) = \bigl(\operatorname{bias}(\hat\theta)\bigr)^2 + \operatorname{var}(\hat\theta)\;} $$
Remember this identity. The bias–variance decomposition is the same statement for a whole model instead of a single number.
for name, est in [("mu_hat", mu_hat), ("sigma_b^2", var_b), ("sigma_u^2", var_u)]:
truth = mu_true if name == "mu_hat" else sigma_true ** 2
bias, var = est.mean() - truth, est.var()
mse = ((est - truth) ** 2).mean()
print(f"{name:10} bias^2 = {bias**2:8.4f} var = {var:8.4f} "
f"sum = {bias**2 + var:8.4f} mse = {mse:8.4f}")
Note $\hat\sigma_b^2$ is biased yet has lower mse than the unbiased $\hat\sigma_u^2$ here.
Unbiasedness is not the goal. Low error is the goal.
Accepting some bias to buy a larger reduction in variance is a good trade.
That sentence is the whole bias–variance trade-off in one line.
A statistic $S$ is sufficient for $\theta$ if, once $S(\mathcal D)$ is known, the rest of the data carries no further information about $\theta$:
$$ p\bigl(\mathcal D \mid S(\mathcal D), \theta\bigr) \;\text{ does not depend on } \theta . $$
All the information about $\theta$ is "compressed" into $S$.
Example: $n$ Bernoulli trials and $S = k = \sum_i X_i$. One particular sequence with $k$ successes has probability $\theta^k(1-\theta)^{n-k}$, and $k$ itself is binomial (Probability), so
$$ p(\mathcal D \mid k, \theta) = \frac{\theta^k(1-\theta)^{n-k}}{\binom nk\,\theta^k(1-\theta)^{n-k}} = \frac{1}{\binom nk} $$
— every ordering is equally likely, whatever $\theta$. To estimate a proportion, $k$ and $n$ are all we need (Uncertainty).
fundamentals-Estimators-exercises.ipynb — pen and paper: bias and variance of the sample mean and of
the sample variance, the mse identity.
fundamentals-Estimators-lab.ipynb — programming: the same estimators over many simulated data sets,
and the mse identity numerically.