Exercises

Machine Learning · Fundamentals · Estimators

Pen-and-paper exercises on estimators: bias and variance of the sample mean and the sample variance, and the mse identity. The programming exercises are in fundamentals-Estimators-lab.ipynb. References point back to the recap Estimators. Solutions are in skip cells (tag solution), stripped for the handout.

Exercise 1 — The sample mean

Let $X_1, \dots, X_n$ be i.i.d. with $\mathbb{E}[X_i] = \mu$ and $\operatorname{var}(X_i) = \sigma^2$, and let $\hat\mu = \bar X = \frac{1}{n}\sum_i X_i$.

Task 1. Show that $\hat\mu$ is unbiased.

Task 2. Show that

$$ \operatorname{var}(\hat\mu) = \frac{\sigma^2}{n} . $$

Hint: use independence, so that $\operatorname{var}(\sum_i X_i) = \sum_i \operatorname{var}(X_i)$.

Task 3. What is $\operatorname{mse}(\hat\mu)$? How fast does the typical error shrink with $n$?

Your answer — Exercise 1

Exercise 2 — The biased sample variance

$$ \hat\sigma_b^2 = \frac{1}{n}\sum_{i=1}^{n}(X_i - \bar X)^2 $$

Task 1. Show that

$$ \mathbb{E}\left[\hat\sigma_b^2\right] = \frac{n-1}{n}\sigma^2, \qquad\text{hence}\qquad \operatorname{bias}(\hat\sigma_b^2) = -\frac{\sigma^2}{n} . $$

Hints.

  • Write $X_i - \bar X = (X_i - \mu) - (\bar X - \mu)$ and expand the square.
  • You will need $\mathbb{E}[(\bar X - \mu)^2] = \sigma^2/n$ from Exercise 1.
  • $\sum_i (X_i - \mu)(\bar X - \mu) = n(\bar X - \mu)^2$.

Task 2. Why is the bias negative — i.e. why does $\hat\sigma_b^2$ systematically underestimate $\sigma^2$?

Answer in terms of $\bar X$ being fitted to the same data.

Task 3. Deduce the correction factor that makes $\hat\sigma_u^2$ unbiased, and explain where the "$n-1$" comes from.

Your answer — Exercise 2

Exercise 3 — The mse identity

Task. Derive, for any estimator $\hat\theta$ of a parameter $\theta$,

$$ \operatorname{mse}(\hat\theta) = \bigl(\operatorname{bias}(\hat\theta)\bigr)^2 + \operatorname{var}(\hat\theta) . $$

Hint: start from $\mathbb{E}[(\theta - \hat\theta)^2]$, expand, and add and subtract $(\mathbb{E}[\hat\theta])^2$ (Estimators).

Follow-up. $\hat\sigma_u^2$ is unbiased and $\hat\sigma_b^2$ is not — yet in the recap (Estimators) $\hat\sigma_b^2$ had the lower mse.

Explain how that is possible, and say in one sentence why "use the unbiased estimator" is not automatically good advice.

Your answer — Exercise 3

Summary

You should be able to answer

  • Why is an estimator a random variable?
  • Why does dividing by $n$ underestimate the variance?
  • Why is an unbiased estimator not automatically the better one?

Literature

  • L. Wasserman: All of Statistics, Springer 2004 — ch. 6
  • I. Goodfellow, Y. Bengio, A. Courville: Deep Learning, MIT Press 2016 — §5.4