Exercises

Machine Learning · L-BiasVar

Pen-and-paper exercises on the bias–variance trade-off. The programming exercises are in the lab notebook lecture-BiasVar-lab.ipynb. References point back to the lecture; the estimator exercises are part of the recap Estimators. Solutions are in skip cells (tag solution), stripped for the handout.

Exercise 1 — Bias and variance of a model

Task 1. State the bias–variance decomposition of $\mathbb{E}_{\mathcal{D}}[E_{out}(h^{\mathcal{D}})]$, and define the average hypothesis $\tilde h$ (lecture).

Task 2. Explain the correspondence with Exercise 3 of the Estimators exercises: what plays the role of $\theta$, of $\hat\theta$, and of the expectation?

Task 3. For each situation, say whether bias or variance dominates:

  1. a constant model $h(x) = b$ fitted to data from $t(x) = \sin(\pi x)$
  2. a degree-15 polynomial fitted to 8 noisy points
  3. a linear model fitted to 10 000 points from a genuinely linear target
  4. a deep neural network trained on 50 images

Task 4. With noisy data a third term appears — the irreducible (Bayes) error.

What is it, and why can no model, however good, reduce it?

Your answer — Exercise 1

Exercise 2 — Errors and model selection

Task. Answer briefly (lecture):

  1. Why does $E_{in}$ fall monotonically with model complexity?
  2. Why does that make $E_{in}$ useless for choosing a model?
  3. Why is the test error only an estimate of $E_{out}$, and what governs its reliability?
  4. In a learning curve with a large, slowly-closing gap between $E_{in}$ and $E_{out}$: would you collect more data, or change the model?

Your answer — Exercise 2

Summary

You should be able to answer

  • Why can $E_{in}$ never select a model?
  • What is the average hypothesis $\tilde h$, and why can it not be computed in practice?
  • Why did the simpler model win in lab Exercise 3?
  • From a learning curve: more data, or a different model?

Literature

  • [Abu] Abu-Mostafa, Magdon-Ismail, Lin: Learning from Data, AMLBook 2012 — §2.3
  • [Has] Hastie, Tibshirani, Friedman: The Elements of Statistical Learning, Springer 2009 — chapter 7
  • C. M. Bishop: Pattern Recognition and Machine Learning, Springer 2006 — §3.2