Machine Learning · Fundamentals · Uncertainty
Pen-and-paper exercises on error bars: a confusion matrix, comparing two estimates, maximum
likelihood for the binomial. The programming exercises are in fundamentals-Uncertainty-lab.ipynb.
References point back to the recap Uncertainty.
Solutions are in skip cells (tag solution), stripped for the handout.
A classifier is evaluated on $200$ test examples:
| predicted 1 | predicted 0 | |
|---|---|---|
| actual 1 | TP = 45 | FN = 15 |
| actual 0 | FP = 5 | TN = 135 |
Task 1. Compute accuracy $\frac{\text{TP}+\text{TN}}{n}$, precision $\frac{\text{TP}}{\text{TP}+\text{FP}}$ and recall $\frac{\text{TP}}{\text{TP}+\text{FN}}$. For each, state $k$ and $n$ (Uncertainty).
Task 2. For each metric compute $\widehat{\operatorname{se}} = \sqrt{\hat\theta(1-\hat\theta)/n}$ and report it in the course convention $\hat\theta \pm 1.96\,\widehat{\operatorname{se}}$, rounded sensibly.
Task 3. A colleague computes the precision's error bar with $n = 200$. What do they get, and why is it wrong?
Two classifiers are evaluated on the same $500$ test examples. Accuracy of A: $0.86$, of B: $0.82$.
Task 1. Treat the two accuracies as independent. Compute both standard errors, the Wald statistic $Z$ and decide at the 5 % level (Uncertainty).
Task 2. Now use that both models saw the same examples. A is right and B wrong on $30$ examples; A wrong and B right on $10$; on all others they agree. With $d^{(i)} = c_A^{(i)} - c_B^{(i)}$, compute $\bar d$, $\operatorname{std}(d)$, $\widehat{\operatorname{se}}(\bar d)$ and the paired $Z$.
Hint: $d^{(i)} \in \{-1, 0, 1\}$, so $\overline{d^2}$ is the share of examples where the two disagree.
Task 3. Why is the paired standard error so much smaller? Answer with $\operatorname{var}(c_A - c_B) = \operatorname{var}(c_A) + \operatorname{var}(c_B) - 2\operatorname{Cov}(c_A, c_B)$.
In $n$ independent trials with success probability $\theta$ we observe $k$ successes: $\mathcal L(\theta) = \binom{n}{k}\theta^k(1-\theta)^{n-k}$.
Task 1. Write down the negative log-likelihood, derive $\hat\theta_{ML} = k/n$, and show that it is a minimum of the NLL (Uncertainty).
Task 2. Let $y^{(i)} \in \{0, 1\}$ be the outcome of trial $i$. Show that, up to a constant, $-\frac{1}{n}\log\mathcal L(\theta)$ is the binary cross-entropy of logistic regression (→ L-LogReg) with the hypothesis $h(\vec x^{(i)}) = \theta$ for every example. What does the constant come from, and why does it not matter?
Task 3. Compute the second derivative of the NLL at $\hat\theta$, and show that its inverse equals $\hat\theta(1-\hat\theta)/n$.