Machine Learning · Fundamentals · Estimators
Pen-and-paper exercises on estimators: bias and variance of the sample mean and the sample
variance, and the mse identity. The programming exercises are in fundamentals-Estimators-lab.ipynb.
References point back to the recap Estimators.
Solutions are in skip cells (tag solution), stripped for the handout.
Let $X_1, \dots, X_n$ be i.i.d. with $\mathbb{E}[X_i] = \mu$ and $\operatorname{var}(X_i) = \sigma^2$, and let $\hat\mu = \bar X = \frac{1}{n}\sum_i X_i$.
Task 1. Show that $\hat\mu$ is unbiased.
Task 2. Show that
$$ \operatorname{var}(\hat\mu) = \frac{\sigma^2}{n} . $$
Hint: use independence, so that $\operatorname{var}(\sum_i X_i) = \sum_i \operatorname{var}(X_i)$.
Task 3. What is $\operatorname{mse}(\hat\mu)$? How fast does the typical error shrink with $n$?
$$ \hat\sigma_b^2 = \frac{1}{n}\sum_{i=1}^{n}(X_i - \bar X)^2 $$
Task 1. Show that
$$ \mathbb{E}\left[\hat\sigma_b^2\right] = \frac{n-1}{n}\sigma^2, \qquad\text{hence}\qquad \operatorname{bias}(\hat\sigma_b^2) = -\frac{\sigma^2}{n} . $$
Hints.
Task 2. Why is the bias negative — i.e. why does $\hat\sigma_b^2$ systematically underestimate $\sigma^2$?
Answer in terms of $\bar X$ being fitted to the same data.
Task 3. Deduce the correction factor that makes $\hat\sigma_u^2$ unbiased, and explain where the "$n-1$" comes from.
Task. Derive, for any estimator $\hat\theta$ of a parameter $\theta$,
$$ \operatorname{mse}(\hat\theta) = \bigl(\operatorname{bias}(\hat\theta)\bigr)^2 + \operatorname{var}(\hat\theta) . $$
Hint: start from $\mathbb{E}[(\theta - \hat\theta)^2]$, expand, and add and subtract $(\mathbb{E}[\hat\theta])^2$ (Estimators).
Follow-up. $\hat\sigma_u^2$ is unbiased and $\hat\sigma_b^2$ is not — yet in the recap (Estimators) $\hat\sigma_b^2$ had the lower mse.
Explain how that is possible, and say in one sentence why "use the unbiased estimator" is not automatically good advice.