Estimators

Machine Learning · Fundamentals

Statistic · estimator · bias · variance · mean squared error

Why estimators?

Every number we report about a model — a test error, an accuracy, a fitted parameter — is computed from a finite, random sample of data. A different sample would have given a different number.

This recap supplies the vocabulary for that — estimator, bias, variance — which the bias–variance trade-off lifts from a number to a whole model. How large the ± of a single estimate is follows in the recap Uncertainty.

Prerequisite: the recap Probability — random variables, expectation, variance, the binomial.

Learning objectives

  1. define a statistic and an estimator, and compute its bias and variance,
  2. prove and use $\;\operatorname{mse}(\hat\theta) = \operatorname{bias}(\hat\theta)^2 + \operatorname{var}(\hat\theta)$,
  3. explain why an unbiased estimator is not automatically the better one,
  4. say what a sufficient statistic is.

Overview

Part The question it answers
1 · Statistic and estimator What is an estimate, and why is it random?
2 · Bias and variance Is an estimator right on average, and how much does it scatter?
3 · The mean squared error How do the two combine into one error?
In [1]:
import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline
np.set_printoptions(precision=4, suppress=True)
plt.rcParams["figure.figsize"] = (9, 4)
plt.rcParams["axes.grid"] = True
plt.rcParams["grid.alpha"] = 0.3
rng = np.random.default_rng(0)

1. Statistic and estimator

A test error, an accuracy, a fitted parameter: each is a number computed from random data. Before we can say how good such a number is, we need one word for it — an estimator — and two words for its quality: bias and variance.

The bias–variance trade-off (→ L-BiasVar) applies exactly the same algebra to whole models.

Data

$$ \mathcal{D} = \{X_1, X_2, \dots, X_n\} $$

Here we regard the data as random variables (the frequentist view), and assume they are i.i.d.:

  • independent — the outcome of one observation does not affect another
  • identically distributed — all $X_i$ are drawn from the same distribution

(Probability)

Statistic and estimator

A statistic is a random variable $S$ that is a function of the data:

$$ S = f(\mathcal{D}) $$

An estimator is a statistic intended to approximate a parameter governing the distribution of $\mathcal{D}$.

Notation: $\hat\theta$ is an estimator of $\theta$.

An estimator is a random variable: it has a distribution, because $\mathcal{D}$ is random. Different data sets give different values of $\hat\theta$.

That is what makes "bias" and "variance" meaningful.

2. Bias and variance

Bias of an estimator

$$ \operatorname{bias}(\hat\theta) := \mathbb{E}\left[\hat\theta\right] - \theta $$

The difference between the mean of the estimator and the true value of $\theta$.

The expectation is taken over different data sets $\mathcal{D}$.

An estimator is unbiased if

$$ \operatorname{bias}(\hat\theta) = 0 \qquad\Longleftrightarrow\qquad \mathbb{E}\left[\hat\theta\right] = \theta . $$

Variance of an estimator

$$ \operatorname{var}(\hat\theta) = \mathbb{E}\left[\left(\hat\theta - \mathbb{E}[\hat\theta]\right)^2\right] = \mathbb{E}[\hat\theta^2] - (\mathbb{E}[\hat\theta])^2 $$

The shortcut uses the rules of variance (Probability): $\mathbb E[\hat\theta]$ is a number, so expanding the square and using linearity gives $\mathbb E[\hat\theta^2] - 2(\mathbb E\hat\theta)^2 + (\mathbb E\hat\theta)^2$.

The expectation is again over data sets $\mathcal D$.

Example: estimating a Gaussian

Data generating distribution:

$$ X_i \sim \mathcal{N}(\mu, \sigma^2) $$

Two parameters to estimate:

  • the mean $\mu = \mathbb{E}[X]$
  • the variance $\sigma^2 = \mathbb{E}[(X - \mu)^2]$

Careful with the word "variance". It is used here for two different things:

  • the variance of the estimator, $\operatorname{var}(\hat\theta)$
  • the variance of the Gaussian, $\sigma^2$ — which is itself the quantity being estimated

Three estimators

Sample mean, an estimator of $\mu$:

$$ \hat\mu = \frac{1}{n}\sum_{i=1}^{n} X_i =: \bar X $$

(Biased) sample variance:

$$ \hat\sigma_b^2 = \frac{1}{n}\sum_{i=1}^{n}(X_i - \bar X)^2 $$

(Unbiased) sample variance:

$$ \hat\sigma_u^2 = \frac{1}{n-1}\sum_{i=1}^{n}(X_i - \bar X)^2 $$

$\hat\mu$ is unbiased

$$ \mathbb{E}\left[\hat\mu\right] = \mathbb{E}\left[\frac{1}{n}\sum_i X_i\right] = \frac{1}{n}\sum_i \mathbb{E}[X_i] = \frac{1}{n}\sum_i \mu = \mu $$

so $\operatorname{bias}(\hat\mu) = 0$.

Variance of $\hat\mu$

The rules of variance (Probability) — a constant factor comes out squared, and for independent $X_i$ the variances add:

$$ \operatorname{var}(\hat\mu) = \operatorname{var}\Bigl(\frac1n \sum_{i=1}^n X_i\Bigr) = \frac{1}{n^2}\operatorname{var}\Bigl(\sum_{i=1}^n X_i\Bigr) = \frac{1}{n^2}\sum_{i=1}^n \operatorname{var}(X_i) = \frac{1}{n^2}\, n\sigma^2 = \frac{\sigma^2}{n} $$

$\operatorname{var}(\hat\mu) = \sigma^2/n$ — the estimator gets more reliable with more data, and its spread shrinks like $1/\sqrt{n}$.

$\hat\sigma_b^2$, in contrast, is biased: $\mathbb{E}[\hat\sigma_b^2] = \frac{n-1}{n}\sigma^2$ (next two slides), i.e. $\operatorname{bias}(\hat\sigma_b^2) = -\sigma^2/n$. It systematically underestimates, because $\bar X$ is fitted to the same data.

Why $\hat\sigma_b^2$ is biased (1/2)

Write each deviation from the fitted mean as $X_i - \bar X = (X_i - \mu) - (\bar X - \mu)$, expand the square, and use $\sum_i (X_i - \mu) = n(\bar X - \mu)$:

$$ \sum_{i}(X_i - \bar X)^2 = \sum_i (X_i - \mu)^2 - 2(\bar X - \mu)\sum_i (X_i - \mu) + n(\bar X - \mu)^2 = \sum_i (X_i - \mu)^2 - n(\bar X - \mu)^2 $$

The deviations from $\bar X$ are smaller than those from $\mu$ — by exactly $n(\bar X - \mu)^2 \ge 0$.

Why $\hat\sigma_b^2$ is biased (2/2)

Take expectations, with $\mathbb E[(X_i - \mu)^2] = \sigma^2$ and $\mathbb E[(\bar X - \mu)^2] = \operatorname{var}(\bar X) = \sigma^2/n$:

$$ \mathbb E\Bigl[\sum_i (X_i - \bar X)^2\Bigr] = n\sigma^2 - \sigma^2 = (n-1)\,\sigma^2 \quad\Longrightarrow\quad \mathbb E[\hat\sigma_b^2] = \frac{n-1}{n}\sigma^2, \qquad \mathbb E[\hat\sigma_u^2] = \sigma^2 $$

Dividing by $n - 1$ instead of $n$ (Bessel's correction) removes the bias.

Let us check that numerically

Draw many data sets, and look at the distribution of each estimator.

In [2]:
n, mu_true, sigma_true, reps = 5, 3.0, 2.0, 100_000
S = rng.normal(mu_true, sigma_true, size=(reps, n))    # one row = one data set D

mu_hat = S.mean(axis=1)
var_b = ((S - mu_hat[:, None]) ** 2).mean(axis=1)          # 1/n
var_u = ((S - mu_hat[:, None]) ** 2).sum(axis=1) / (n - 1)  # 1/(n-1)
print(f"{reps} data sets of n = {n}")
100000 data sets of n = 5
In [3]:
print(f"E[mu_hat]     = {mu_hat.mean():.4f}   bias = {mu_hat.mean() - mu_true:+.4f}  (theory 0)")
print(f"var(mu_hat)   = {mu_hat.var():.4f}   theory sigma^2/n = {sigma_true**2 / n:.4f}")
E[mu_hat]     = 3.0034   bias = +0.0034  (theory 0)
var(mu_hat)   = 0.7997   theory sigma^2/n = 0.8000
In [4]:
print(f"E[sigma_b^2]  = {var_b.mean():.4f}   bias = {var_b.mean() - sigma_true**2:+.4f}"
      f"   (theory {-sigma_true**2 / n:+.4f})")
print(f"E[sigma_u^2]  = {var_u.mean():.4f}   bias = {var_u.mean() - sigma_true**2:+.4f}  (theory 0)")
E[sigma_b^2]  = 3.2097   bias = -0.7903   (theory -0.8000)
E[sigma_u^2]  = 4.0121   bias = +0.0121  (theory 0)
In [5]:
fig, ax = plt.subplots(1, 2, figsize=(11, 3.6))
ax[0].hist(mu_hat, bins=60, color="steelblue")
ax[0].axvline(mu_true, color="k", lw=2, label=r"true $\mu$")
ax[0].set_title(r"distribution of $\hat\mu$"); ax[0].legend()
ax[1].hist(var_b, bins=60, range=(0, 20), color="indianred", alpha=.7, label=r"$\hat\sigma_b^2$")
ax[1].hist(var_u, bins=60, range=(0, 20), color="seagreen", alpha=.5, label=r"$\hat\sigma_u^2$")
ax[1].axvline(sigma_true**2, color="k", lw=2, label=r"true $\sigma^2$")
ax[1].set_title("distribution of the variance estimators"); ax[1].legend()
plt.tight_layout(); plt.show()
No description has been provided for this image

3. The mean squared error

Mean squared error of an estimator

$$ \operatorname{mse}(\hat\theta) = \mathbb{E}\left[\left(\theta - \hat\theta\right)^2\right] $$

This is what we actually care about: how far the estimate lands from the truth, on average.

The decomposition

$\theta$ is a constant, so linearity (Probability) gives the second line; then add and subtract $(\mathbb E[\hat\theta])^2$:

$$ \begin{aligned} \operatorname{mse}(\hat\theta) &= \mathbb{E}\left[(\theta - \hat\theta)^2\right] \\ &= \theta^2 - 2\theta\,\mathbb{E}[\hat\theta] + \mathbb{E}[\hat\theta^2] \;\underbrace{+\; (\mathbb{E}[\hat\theta])^2 - (\mathbb{E}[\hat\theta])^2}_{= \,0} \end{aligned} $$

Regrouping the six terms into two brackets:

$$ \begin{aligned} &= \underbrace{\left(\theta^2 - 2\theta\,\mathbb{E}[\hat\theta] + (\mathbb{E}[\hat\theta])^2\right)}_{(\mathbb{E}[\hat\theta] - \theta)^2} + \underbrace{\left(\mathbb{E}[\hat\theta^2] - (\mathbb{E}[\hat\theta])^2\right)}_{\operatorname{var}(\hat\theta)} \end{aligned} $$

$$ \boxed{\;\operatorname{mse}(\hat\theta) = \bigl(\operatorname{bias}(\hat\theta)\bigr)^2 + \operatorname{var}(\hat\theta)\;} $$

Remember this identity. The bias–variance decomposition is the same statement for a whole model instead of a single number.

In [6]:
for name, est in [("mu_hat", mu_hat), ("sigma_b^2", var_b), ("sigma_u^2", var_u)]:
    truth = mu_true if name == "mu_hat" else sigma_true ** 2
    bias, var = est.mean() - truth, est.var()
    mse = ((est - truth) ** 2).mean()
    print(f"{name:10}  bias^2 = {bias**2:8.4f}  var = {var:8.4f}  "
          f"sum = {bias**2 + var:8.4f}  mse = {mse:8.4f}")
mu_hat      bias^2 =   0.0000  var =   0.7997  sum =   0.7997  mse =   0.7997
sigma_b^2   bias^2 =   0.6246  var =   5.1756  sum =   5.8002  mse =   5.8002
sigma_u^2   bias^2 =   0.0001  var =   8.0869  sum =   8.0870  mse =   8.0870

Note $\hat\sigma_b^2$ is biased yet has lower mse than the unbiased $\hat\sigma_u^2$ here.

Unbiasedness is not the goal. Low error is the goal.

Accepting some bias to buy a larger reduction in variance is a good trade.

That sentence is the whole bias–variance trade-off in one line.

Sufficient statistics

A statistic $S$ is sufficient for $\theta$ if, once $S(\mathcal D)$ is known, the rest of the data carries no further information about $\theta$:

$$ p\bigl(\mathcal D \mid S(\mathcal D), \theta\bigr) \;\text{ does not depend on } \theta . $$

All the information about $\theta$ is "compressed" into $S$.

Example: $n$ Bernoulli trials and $S = k = \sum_i X_i$. One particular sequence with $k$ successes has probability $\theta^k(1-\theta)^{n-k}$, and $k$ itself is binomial (Probability), so

$$ p(\mathcal D \mid k, \theta) = \frac{\theta^k(1-\theta)^{n-k}}{\binom nk\,\theta^k(1-\theta)^{n-k}} = \frac{1}{\binom nk} $$

— every ordering is equally likely, whatever $\theta$. To estimate a proportion, $k$ and $n$ are all we need (Uncertainty).

4. Summary

Take away

  1. An estimator is a random variable: it has a bias and a variance, and $\operatorname{mse} = \operatorname{bias}^2 + \operatorname{var}$.
  2. Unbiased is not the goal — low error is.
  3. The sample mean is unbiased with variance $\sigma^2/n$: its scatter shrinks like $1/\sqrt n$.

Exercises and Lab

fundamentals-Estimators-exercises.ipynb — pen and paper: bias and variance of the sample mean and of the sample variance, the mse identity.

fundamentals-Estimators-lab.ipynb — programming: the same estimators over many simulated data sets, and the mse identity numerically.

Literature

  • L. Wasserman: All of Statistics, Springer 2004 — ch. 6 (estimators). doi:10.1007/978-0-387-21736-9
  • I. Goodfellow, Y. Bengio, A. Courville: Deep Learning, MIT Press 2016 — §5.4 (estimators, bias, variance).