Exercises

Machine Learning · Fundamentals · Uncertainty

Pen-and-paper exercises on error bars: a confusion matrix, comparing two estimates, maximum likelihood for the binomial. The programming exercises are in fundamentals-Uncertainty-lab.ipynb. References point back to the recap Uncertainty. Solutions are in skip cells (tag solution), stripped for the handout.

Exercise 1 — Error bars from a confusion matrix

A classifier is evaluated on $200$ test examples:

predicted 1 predicted 0
actual 1 TP = 45 FN = 15
actual 0 FP = 5 TN = 135

Task 1. Compute accuracy $\frac{\text{TP}+\text{TN}}{n}$, precision $\frac{\text{TP}}{\text{TP}+\text{FP}}$ and recall $\frac{\text{TP}}{\text{TP}+\text{FN}}$. For each, state $k$ and $n$ (Uncertainty).

Task 2. For each metric compute $\widehat{\operatorname{se}} = \sqrt{\hat\theta(1-\hat\theta)/n}$ and report it in the course convention $\hat\theta \pm 1.96\,\widehat{\operatorname{se}}$, rounded sensibly.

Task 3. A colleague computes the precision's error bar with $n = 200$. What do they get, and why is it wrong?

Your answer — Exercise 1

Exercise 2 — Is the difference real?

Two classifiers are evaluated on the same $500$ test examples. Accuracy of A: $0.86$, of B: $0.82$.

Task 1. Treat the two accuracies as independent. Compute both standard errors, the Wald statistic $Z$ and decide at the 5 % level (Uncertainty).

Task 2. Now use that both models saw the same examples. A is right and B wrong on $30$ examples; A wrong and B right on $10$; on all others they agree. With $d^{(i)} = c_A^{(i)} - c_B^{(i)}$, compute $\bar d$, $\operatorname{std}(d)$, $\widehat{\operatorname{se}}(\bar d)$ and the paired $Z$.

Hint: $d^{(i)} \in \{-1, 0, 1\}$, so $\overline{d^2}$ is the share of examples where the two disagree.

Task 3. Why is the paired standard error so much smaller? Answer with $\operatorname{var}(c_A - c_B) = \operatorname{var}(c_A) + \operatorname{var}(c_B) - 2\operatorname{Cov}(c_A, c_B)$.

Your answer — Exercise 2

Exercise 3 — Maximum likelihood for the binomial

In $n$ independent trials with success probability $\theta$ we observe $k$ successes: $\mathcal L(\theta) = \binom{n}{k}\theta^k(1-\theta)^{n-k}$.

Task 1. Write down the negative log-likelihood, derive $\hat\theta_{ML} = k/n$, and show that it is a minimum of the NLL (Uncertainty).

Task 2. Let $y^{(i)} \in \{0, 1\}$ be the outcome of trial $i$. Show that, up to a constant, $-\frac{1}{n}\log\mathcal L(\theta)$ is the binary cross-entropy of logistic regression (→ L-LogReg) with the hypothesis $h(\vec x^{(i)}) = \theta$ for every example. What does the constant come from, and why does it not matter?

Task 3. Compute the second derivative of the NLL at $\hat\theta$, and show that its inverse equals $\hat\theta(1-\hat\theta)/n$.

Your answer — Exercise 3

Summary

You should be able to answer

  • What is the $n$ in the error bar of a precision? Of a recall?
  • What does "95 %" in a confidence interval refer to?
  • When do you need the bootstrap, and what does it not fix?
  • Why must two models on the same test set be compared paired?
  • Why is minimising the cross-entropy a maximum-likelihood estimate?

Literature

  • L. Wasserman: All of Statistics, Springer 2004 — ch. 6, ch. 8, §9.7, §10.1
  • I. Goodfellow, Y. Bengio, A. Courville: Deep Learning, MIT Press 2016 — §5.4, §5.5