Machine Learning · Fundamentals
Error bars for estimates: standard errors · confidence intervals · maximum likelihood · the bootstrap · comparing two models
Every number we report about a model is computed from a finite, random sample of data. A different test set would have given a different number. So
accuracy = 0.92 is not a result. accuracy = 0.92 ± 0.03 is.
This recap supplies the ±; the evaluation metrics carry it.
Prerequisite: the recaps Probability (random variables, expectation, variance, the binomial) and Estimators (estimator, bias, variance).
| Part | The question it answers |
|---|---|
| 1 · Standard error | How large is the uncertainty of one estimate? |
| 2 · Confidence intervals | What exactly does "± 0.06" promise? |
| 3 · Maximum likelihood | Where does the estimator $k/n$ come from — and the cross-entropy? |
| 4 · Proportions | How large is the error bar of an accuracy — or of any other rate? |
| 5 · The bootstrap | And for a metric without a formula? |
| 6 · Comparing two estimates | Is model A really better than model B? |
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
np.set_printoptions(precision=4, suppress=True)
plt.rcParams["figure.figsize"] = (9, 4)
plt.rcParams["axes.grid"] = True
plt.rcParams["grid.alpha"] = 0.3
rng = np.random.default_rng(0)
The variance tells us how much $\hat\theta$ scatters over many data sets.
But we only ever have one data set. What can it tell us about its own uncertainty?
The standard error of an estimator is its standard deviation:
$$ \operatorname{se}(\hat\theta) = \sqrt{\operatorname{var}(\hat\theta)} $$
It has the same unit as $\theta$ — which is why we report it, not the variance.
For the sample mean (Estimators): $\;\operatorname{se}(\hat\mu) = \dfrac{\sigma}{\sqrt n}$.
$\sigma$ is unknown — but it can itself be estimated from the same data:
$$ \widehat{\operatorname{se}}(\hat\mu) = \frac{\hat\sigma_u}{\sqrt n} $$
One data set gives both the estimate and its error bar.
That is the whole trick, and it works because the formula for $\operatorname{se}$ was derived before looking at the data.
mu_true, sigma_true = 3.0, 2.0 # the Gaussian of Estimators §2
x = rng.normal(mu_true, sigma_true, size=25) # ONE data set, n = 25
mu_hat_1 = x.mean()
se_hat_1 = x.std(ddof=1) / np.sqrt(len(x)) # sigma_u / sqrt(n)
print(f"mu = {mu_hat_1:.2f} ± {se_hat_1:.2f} (true mu = {mu_true}, "
f"true se = {sigma_true / 5:.2f})")
Does the error bar from one data set describe the scatter over many?
S25 = rng.normal(mu_true, sigma_true, size=(20_000, 25)) # 20 000 data sets
means = S25.mean(axis=1)
se_hats = S25.std(axis=1, ddof=1) / 5
print(f"std of the 20 000 estimates : {means.std():.4f}")
print(f"theory sigma / sqrt(n) : {sigma_true / 5:.4f}")
print(f"typical se_hat from ONE set : {se_hats.mean():.4f}")
$$ \text{result} = (\text{best estimate} \pm \text{uncertainty})\;\text{unit} $$
Without an uncertainty, an estimate cannot be compared with anything — not with another model, not with last year's result.
After S. Bell, A Beginner's Guide to Uncertainty of Measurement, NPL 2001.
For large $n$, a sample mean is approximately Gaussian — whatever the distribution of the individual $X_i$:
$$ \hat\mu \;\dot\sim\; \mathcal{N}\!\left(\mu,\; \operatorname{se}(\hat\mu)^2\right) $$
Many estimators behave the same way, in particular every proportion (§4).
Stated here, not proven (Wasserman 2004, Thm. 5.8); the figure below shows it.
fig, ax = plt.subplots(1, 3, figsize=(12, 3.2))
for a, n_ in zip(ax, [1, 5, 30]):
m = rng.exponential(1.0, size=(50_000, n_)).mean(axis=1) # skewed data
a.hist(m, bins=80, density=True, color="steelblue")
a.set_title(f"mean of n = {n_} exponential draws")
plt.tight_layout(); plt.show()
95 % of a Gaussian lies within $1.96$ standard deviations of its mean (Probability). With $\hat\theta \;\dot\sim\; \mathcal N(\theta, \operatorname{se}^2)$:
$$ p\bigl(|\hat\theta - \theta| \le 1.96\,\operatorname{se}\bigr) \approx 0.95 $$
The event $|\hat\theta - \theta| \le 1.96\,\operatorname{se}$ is the same as $\hat\theta - 1.96\,\operatorname{se} \le \theta \le \hat\theta + 1.96\,\operatorname{se}$ — an interval around $\hat\theta$ that contains $\theta$ with probability $\approx 0.95$. In practice $\operatorname{se}$ is replaced by $\widehat{\operatorname{se}}$.
Careful. $\theta$ is fixed; the interval is random. "95 %" is a property of the procedure: of many intervals built this way, about 95 % contain $\theta$.
It is not "θ lies in this interval with probability 95 %".
Both conventions are in use, so always say which one you mean:
| written | half-width | coverage | common in |
|---|---|---|---|
| $\hat\theta \pm \widehat{\operatorname{se}}$ | 1 se | ≈ 68 % | physics, measurement |
| $\hat\theta \pm 1.96\,\widehat{\operatorname{se}}$ | 1.96 se | ≈ 95 % | statistics, medicine |
Convention in this course: $a \pm b$ means a 95 % confidence interval, $b = 1.96\,\widehat{\operatorname{se}}$ — unless stated otherwise.
def ci_covers(S, mu, z):
'''For every row of S: does mean ± z * se_hat contain mu?'''
m = S.mean(axis=1)
se = S.std(axis=1, ddof=1) / np.sqrt(S.shape[1])
return np.abs(m - mu) <= z * se
for z in [1.0, 1.96]:
print(f"mean ± {z:4} se_hat contains mu in {ci_covers(S25, mu_true, z).mean():.1%}"
" of 20 000 data sets")
cov = ci_covers(S25[:50], mu_true, 1.96)
lo = means[:50] - 1.96 * se_hats[:50]; hi = means[:50] + 1.96 * se_hats[:50]
fig, ax = plt.subplots(figsize=(11, 3.4))
for i in range(50):
ax.plot([i, i], [lo[i], hi[i]], color="steelblue" if cov[i] else "red", lw=2)
ax.axhline(mu_true, color="k", lw=1.5, label=r"true $\mu$")
ax.set_xlabel("data set"); ax.set_title("50 data sets, 50 intervals (red: misses $\\mu$)")
ax.legend(); plt.show()
$1.96$ assumes $\operatorname{se}$ is known. With $\widehat{\operatorname{se}}$ from very few points the interval must be wider: use the $t$-distribution with $n-1$ degrees of freedom (for Gaussian data).
| $n$ | 2 | 3 | 5 | 10 | 20 | 30 | $\infty$ |
|---|---|---|---|---|---|---|---|
| multiplier | 12.71 | 4.30 | 2.78 | 2.26 | 2.09 | 2.05 | 1.96 |
This is why the simulation above, with $n = 25$ and $1.96$, covered slightly less than 95 %. Above $n \approx 30$ the difference is negligible — and test sets in ML are far larger.
So far the estimators were simply given: $\bar X$, $\hat\sigma_b^2$, $\hat\sigma_u^2$ (Estimators).
Where do estimators come from? The most important recipe is maximum likelihood. Logistic regression (→ L-LogReg) uses it to derive the cross-entropy; here is its simplest case — a proportion, such as the accuracy of a classifier.
A model $p(\mathcal D \mid \theta)$ gives the probability of the data for each value of the parameter. Read with the data fixed and $\theta$ varying, it is the likelihood:
$$ \mathcal L(\theta) = p(\mathcal D \mid \theta) $$
"Never say 'the likelihood of the data'. Always say 'the likelihood of the parameters'. The likelihood function is not a probability distribution."
D. MacKay, Information Theory, Inference and Learning Algorithms, 2003, p. 29
It need not integrate to 1 over $\theta$.
Choose the parameter under which the observed data are most probable:
$$ \hat\theta_{ML} = \arg\max_\theta \mathcal L(\theta) = \arg\min_\theta \bigl(-\log \mathcal L(\theta)\bigr) $$
The logarithm is monotonic, so the location of the optimum does not move — but products become sums, which are easier to differentiate and numerically stable. $-\log\mathcal L$ is the negative log-likelihood (NLL).
A classifier is applied to $n$ independent test examples. Each prediction is correct with the same unknown probability $\theta$. We observe $k$ correct predictions.
The number of correct predictions is binomially distributed (Probability):
$$ \mathcal L(\theta) = p(k \mid n, \theta) = \binom{n}{k}\,\theta^{k}(1-\theta)^{n-k} $$
def binom_nll(theta, k, n):
'''Negative log-likelihood of the binomial, without the constant log C(n, k).'''
return -(k * np.log(theta) + (n - k) * np.log(1 - theta))
th = np.linspace(0.4, 0.95, 500)
fig, ax = plt.subplots(1, 2, figsize=(11, 3.4))
for k_, n_ in [(7, 10), (70, 100)]:
L = np.exp(-(binom_nll(th, k_, n_) - binom_nll(k_ / n_, k_, n_))) # max scaled to 1
ax[0].plot(th, L, label=f"k = {k_}, n = {n_}")
ax[1].plot(th, binom_nll(th, k_, n_) / n_, label=f"k = {k_}, n = {n_}")
ax[0].set_title(r"likelihood $\mathcal{L}(\theta)$, scaled to max 1")
ax[1].set_title(r"mean NLL $-\frac{1}{n}\log\mathcal{L}(\theta)$")
for a in ax: a.axvline(0.7, color="k", ls=":"); a.set_xlabel(r"$\theta$"); a.legend()
plt.tight_layout(); plt.show()
Both curves peak at $\theta = 0.7$ — but the one built on ten times more data is much narrower. Keep that picture in mind: it will turn out to be the error bar.
Take the negative logarithm; the binomial coefficient does not depend on $\theta$:
$$ -\log\mathcal L(\theta) = -\log\binom{n}{k} \;-\; k\log\theta \;-\; (n-k)\log(1-\theta) $$
Set the derivative to zero — with $\frac{d}{d\theta}\log\theta = \frac1\theta$ and, by the chain rule, $\frac{d}{d\theta}\log(1-\theta) = -\frac{1}{1-\theta}$:
$$ 0 = \frac{d}{d\theta}\bigl(-\log\mathcal L\bigr) = -\frac{k}{\theta} + \frac{n-k}{1-\theta} $$
Multiply by $\theta(1-\theta)$:
$$ k(1-\theta) = (n-k)\,\theta \quad\Longrightarrow\quad k = n\theta \quad\Longrightarrow\quad \boxed{\;\hat\theta_{ML} = \frac{k}{n}\;} $$
Differentiating once more gives $\frac{k}{\theta^2} + \frac{n-k}{(1-\theta)^2} > 0$ for every $\theta \in (0, 1)$: the NLL is convex, so $k/n$ is its minimum.
The maximum-likelihood estimate of a probability is the observed frequency. An accuracy is an estimate of exactly this kind — and so is every other rate of a classifier that counts "$k$ out of $n$" (§4).
Evaluate the NLL on a fine grid and look where the minimum is:
k_obs, n_obs = 88, 100
grid = np.linspace(0.001, 0.999, 99_801)
print(f"grid minimum of the NLL at theta = {grid[np.argmin(binom_nll(grid, k_obs, n_obs))]:.4f}")
print(f"k / n = {k_obs / n_obs:.4f}")
Write the data example by example: $y^{(i)} = 1$ if prediction $i$ is correct, else $0$, so $k = \sum_i y^{(i)}$. One example has $p(y^{(i)} \mid \theta) = \theta^{y^{(i)}}(1-\theta)^{1-y^{(i)}}$ — that is $\theta$ for $y = 1$ and $1 - \theta$ for $y = 0$. The examples are independent, so their probabilities multiply:
$$ \prod_{i=1}^n \theta^{y^{(i)}}(1-\theta)^{1-y^{(i)}} = \theta^{k}(1-\theta)^{n-k} $$
— the binomial likelihood without $\binom nk$, which only counts the orderings (a constant).
The log turns the product into a sum, so the mean NLL is
$$ -\frac{1}{n}\log\mathcal L(\theta) = -\frac{1}{n}\sum_{i=1}^{n}\Bigl[\,y^{(i)}\log\theta + \bigl(1-y^{(i)}\bigr)\log(1-\theta)\Bigr] \;(+\text{const}) $$
Compare the cost of logistic regression (→ L-LogReg):
$$ J(\vec\theta) = -\frac{1}{m}\sum_{i=1}^{m}\Bigl[\,y^{(i)}\log h_\theta(\vec x^{(i)}) + \bigl(1-y^{(i)}\bigr)\log\bigl(1-h_\theta(\vec x^{(i)})\bigr)\Bigr] $$
The same expression. Here the probability is one constant $\theta$ for every example; in logistic regression each example gets its own, $h_\theta(\vec x^{(i)}) = g(\vec\theta^{\,T}\vec x^{(i)})$, and the parameters are the weights inside $g$.
def cross_entropy(y, h):
'''The binary cross-entropy of logistic regression, for labels y and predicted probabilities h.'''
return -np.mean(y * np.log(h) + (1 - y) * np.log(1 - h))
y_obs = np.r_[np.ones(k_obs), np.zeros(n_obs - k_obs)] # 88 correct, 12 wrong
for theta in [0.7, 0.88]:
print(f"theta = {theta}: cross-entropy = {cross_entropy(y_obs, theta):.5f} "
f"NLL / n = {binom_nll(theta, k_obs, n_obs) / n_obs:.5f}")
Minimising the cross-entropy is maximum-likelihood estimation — for a constant probability it returns $k/n$, for a logistic model it returns the weights $\vec\theta$.
Evaluate the second derivative of the NLL at its minimum $\hat\theta = k/n$, i.e. with $k = n\hat\theta$ and $n - k = n(1 - \hat\theta)$:
$$ \frac{d^2}{d\theta^2}\bigl(-\log\mathcal L\bigr)\Big|_{\hat\theta} = \frac{n\hat\theta}{\hat\theta^2} + \frac{n(1-\hat\theta)}{(1-\hat\theta)^2} = \frac{n}{\hat\theta} + \frac{n}{1-\hat\theta} = \frac{n}{\hat\theta(1-\hat\theta)} $$
Its inverse, $\hat\theta(1-\hat\theta)/n$, is exactly the variance of $\hat\theta$ that §4 derives directly.
This is no coincidence: in general, for large $n$,
$$ \widehat{\operatorname{se}}(\hat\theta_{ML}) \approx \Bigl(\tfrac{d^2}{d\theta^2}(-\log\mathcal L)\big|_{\hat\theta_{ML}}\Bigr)^{-1/2} $$
(the Fisher information; Wasserman 2004, §9.7 — stated here, not proven)
A sharper likelihood means a smaller error bar — the picture above.
eps = 1e-4 # finite-difference step
t0 = k_obs / n_obs
curv = (binom_nll(t0 + eps, k_obs, n_obs) - 2 * binom_nll(t0, k_obs, n_obs)
+ binom_nll(t0 - eps, k_obs, n_obs)) / eps**2
print(f"se from the curvature : {1 / np.sqrt(curv):.4f}")
print(f"sqrt(theta(1-theta)/n): {np.sqrt(t0 * (1 - t0) / n_obs):.4f}")
Let $X_i \in \{0, 1\}$ with $p(X_i = 1) = \theta$ — "prediction $i$ is correct". Then
$$ \hat\theta = \frac{k}{n} = \frac{1}{n}\sum_{i=1}^{n} X_i $$
is a sample mean — so everything from §1 applies, and so does Estimators.
Since $X_i^2 = X_i$: $\;\mathbb{E}[X_i] = \theta$, $\;\operatorname{var}(X_i) = \mathbb{E}[X_i^2] - \theta^2 = \theta(1-\theta)$ (Probability).
With $\operatorname{var}(\bar X) = \sigma^2/n$ from Estimators:
$$ \mathbb{E}[\hat\theta] = \theta, \qquad \operatorname{var}(\hat\theta) = \frac{\theta(1-\theta)}{n}, \qquad \widehat{\operatorname{se}}(\hat\theta) = \sqrt{\frac{\hat\theta(1-\hat\theta)}{n}} $$
The error bar of a proportion needs no extra data: $\hat\theta$ and $n$ are enough.
Its size: $\theta(1-\theta)$ is largest at $\theta = 0.5$ (Probability), so at worst $\pm 1.96\sqrt{0.5 \cdot 0.5/n} \approx \pm 1/\sqrt n$ — about $\pm 0.10$ for $n = 100$, $\pm 0.03$ for $n = 1000$.
For a binary classifier, every test example falls into one of four groups — the confusion matrix:
| predicted 1 | predicted 0 | |
|---|---|---|
| actual 1 | true positive (TP) | false negative (FN) |
| actual 0 | false positive (FP) | true negative (TN) |
Accuracy is $(\text{TP} + \text{TN})/n$. It hides which mistakes are made.
Two further rates, each a conditional probability (Probability):
Why these are needed, and further rates, are the subject of → L-Eval. Here they only need error bars.
A classifier evaluated on $400$ test examples:
| predicted 1 | predicted 0 | |
|---|---|---|
| actual 1 | TP = 88 | FN = 22 |
| actual 0 | FP = 12 | TN = 278 |
| metric | $k$ | $n$ = the examples it is computed on |
|---|---|---|
| accuracy | TP + TN | all examples, 400 |
| precision | TP | predicted positives, TP + FP = 100 |
| recall | TP | actual positives, TP + FN = 110 |
def proportion_ci(k, n, z=1.96):
'''Estimate k/n and the half-width z * se_hat of its confidence interval.'''
theta_hat = k / n
return theta_hat, z * np.sqrt(theta_hat * (1 - theta_hat) / n)
TP, FN, FP, TN = 88, 22, 12, 278
for name, k_, n_ in [("accuracy", TP + TN, TP + FN + FP + TN),
("precision", TP, TP + FP), ("recall", TP, TP + FN)]:
est, half = proportion_ci(k_, n_)
print(f"{name:9} = {est:.2f} ± {half:.2f} (95 % CI, n = {n_})")
The common mistake: computing the precision's error bar with $n = 400$. That gives $\pm 0.03$ — too narrow by the factor $\sqrt{400/100} = 2$. Only the 100 examples predicted positive say anything about precision; the other 300 carry no information about it.
$\hat\theta \pm 1.96\,\widehat{\operatorname{se}}$ relies on the central limit theorem. It breaks down when $n$ is small or $\hat\theta$ is near 0 or 1: with $10$ correct out of $10$, $\widehat{\operatorname{se}} = 0$ and the "interval" is the single point $1$.
Rule of thumb: use it only if $n\hat\theta \ge 5$ and $n(1-\hat\theta) \ge 5$. Otherwise use the Wilson interval below, or the bootstrap of §5 with care.
The Wilson score interval (with $z = 1.96$) has centre and half-width
$$ \frac{\hat\theta + \frac{z^2}{2n}}{1 + \frac{z^2}{n}} \qquad\text{and}\qquad \frac{z}{1 + \frac{z^2}{n}}\sqrt{\frac{\hat\theta(1-\hat\theta)}{n} + \frac{z^2}{4n^2}} $$
For 10 out of 10 it gives $[0.72,\, 1.00]$ — an honest statement that ten trials are not many.
Precision and recall have a formula for their error bar. Many metrics do not — for example the $F_1$ score, which combines precision and recall into one number (their harmonic mean; the reason is given in → L-Eval):
$$ F_1 = \frac{2\cdot\text{precision}\cdot\text{recall}}{\text{precision}+\text{recall}} $$
— or the median. Ideally we would draw many new test sets and look at the spread of the metric — the simulations of Estimators. We cannot: we have one test set.
Replace the unknown distribution of the data by the empirical distribution of the $n$ points we have — and simulate from that.
Drawing from the empirical distribution = drawing $n$ points from our data set with replacement. Each such resample is a plausible "other test set".
B. Efron, 1979. Wasserman 2004, ch. 8.
For $b = 1, \dots, B$ (typically $B = 1000$ to $10\,000$):
Then
def bootstrap(stat, data, B=5000, rng=rng):
'''B bootstrap replicates of stat(data); data has one example per row.'''
n = len(data)
reps = np.empty(B)
for b in range(B):
idx = rng.integers(0, n, size=n) # n draws WITH replacement
reps[b] = stat(data[idx])
return reps
Sanity check on a case with a formula — the precision of §4: 88 ones, 12 zeros.
correct = np.r_[np.ones(88), np.zeros(12)]
reps = bootstrap(np.mean, correct)
print(f"bootstrap se : {reps.std():.4f}")
print(f"formula se : {np.sqrt(0.88 * 0.12 / 100):.4f}")
Resample what was sampled: whole test examples, i.e. pairs $(y, \hat y)$ — not the four cells of the confusion matrix independently.
def f1(pairs):
'''F1 from an array of (y, y_hat) rows.'''
y, yh = pairs[:, 0], pairs[:, 1]
tp = np.sum((y == 1) & (yh == 1))
return 2 * tp / (2 * tp + np.sum((y == 0) & (yh == 1)) + np.sum((y == 1) & (yh == 0)))
pairs = np.array([[1, 1]] * TP + [[1, 0]] * FN + [[0, 1]] * FP + [[0, 0]] * TN)
reps_f1 = bootstrap(f1, pairs)
lo, hi = np.quantile(reps_f1, [0.025, 0.975])
print(f"F1 = {f1(pairs):.2f} ± {1.96 * reps_f1.std():.2f} "
f"(95 % CI from 1.96 se_boot)")
print(f"percentile interval: [{lo:.2f}, {hi:.2f}]")
fig, ax = plt.subplots(figsize=(8, 3))
ax.hist(reps_f1, bins=60, color="steelblue")
ax.axvline(f1(pairs), color="k", lw=2, label="F1 on the test set")
for q in (lo, hi): ax.axvline(q, color="red", ls="--")
ax.set_title("5000 bootstrap replicates of $F_1$"); ax.legend(); plt.show()
The bootstrap estimates the variance due to the finite test set. It cannot repair a test set that is too small, unrepresentative, or has been used to choose the model — those are bias problems.
Model A: precision $0.88 \pm 0.06$. Model B: precision $0.81 \pm 0.07$.
Is A better — or is the difference just test-set noise?
For two independent estimates with 95 % bars of similar size:
Overlap means "not shown", never "shown to be equal".
For independent estimates the covariance vanishes and the variances of the difference add (Probability): $\operatorname{var}(\hat\theta_A - \hat\theta_B) = \operatorname{se}_A^2 + \operatorname{se}_B^2$. Dividing the difference by its standard error gives variance 1 (Rule 1), and by the CLT it is near-Gaussian. If the true values are equal,
$$ Z = \frac{\hat\theta_A - \hat\theta_B}{\sqrt{\widehat{\operatorname{se}}_A^2 + \widehat{\operatorname{se}}_B^2}} \;\dot\sim\; \mathcal N(0, 1) $$
so $|Z| > 1.96$ happens by chance only 5 % of the time. Wasserman 2004, §10.1.
This is where the overlap rules come from. Take equal standard errors, so the difference has standard error $\sqrt2\,\operatorname{se}$:
Bars may overlap a little while the difference is significant.
n_sim, sig = 10, 1.0
se = sig / np.sqrt(n_sim)
A = rng.normal(0.0, sig, (100_000, n_sim)).mean(axis=1) # two estimates of the
B = rng.normal(0.0, sig, (100_000, n_sim)).mean(axis=1) # SAME true value 0
gap = np.abs(A - B) / se
print(f"Wald |Z| > 1.96 : {np.mean(gap / np.sqrt(2) > 1.96):.3f} (target 0.05)")
print(f"95 % bars do not touch : {np.mean(gap > 2 * 1.96):.3f}")
print(f"overlap by half an arm : {np.mean(gap > 1.5 * 1.96):.3f}")
Two models evaluated on the same test set are not independent: both tend to get the easy examples right and the hard ones wrong.
Use the per-example difference of the correctness indicators, $d^{(i)} = c_A^{(i)} - c_B^{(i)} \in \{-1, 0, 1\}$:
$$ \hat\theta_A - \hat\theta_B = \bar d, \qquad \widehat{\operatorname{se}}(\bar d) = \frac{\operatorname{std}(d)}{\sqrt n} $$
$\bar d$ is a sample mean, so its standard error is that of §4. It is small because $\operatorname{var}(d) = \operatorname{var}(c_A) + \operatorname{var}(c_B) - 2\operatorname{Cov}(c_A, c_B)$ (Probability), and the covariance is positive: examples both models get right (or both wrong) contribute $d = 0$ — they cancel.
def two_classifiers(n, acc_a, acc_b, rng, redraw=0.2):
'''Correctness of A and B on the same n examples, sharing example difficulty.'''
u = rng.random(n) # difficulty
u_b = np.where(rng.random(n) < redraw, rng.random(n), u) # B differs on 20 %
return (u < acc_a).astype(float), (u_b < acc_b).astype(float)
def z_independent(ca, cb):
a, b, n = ca.mean(), cb.mean(), len(ca)
return (a - b) / np.sqrt(a * (1 - a) / n + b * (1 - b) / n)
def z_paired(ca, cb):
d = ca - cb
return d.mean() / (d.std(ddof=1) / np.sqrt(len(d)))
sim_rng = np.random.default_rng(1)
z_i, z_p = [], []
for _ in range(2000): # 2000 test sets, n = 500
ca, cb = two_classifiers(500, 0.86, 0.82, sim_rng)
z_i.append(z_independent(ca, cb)); z_p.append(z_paired(ca, cb))
print("A is truly 4 points better. Share of test sets where the test detects it:")
print(f" independent Wald : {np.mean(np.array(z_i) > 1.96):.2f}")
print(f" paired : {np.mean(np.array(z_p) > 1.96):.2f}")
On a shared test set, always compare paired. The independent test throws away the fact that both models saw the same examples, and misses real differences.
The bootstrap version: resample example indices once, compute both metrics on that resample, and look at the spread of their difference (a paired bootstrap), as used in → L-Eval.
fundamentals-Uncertainty-exercises.ipynb — pen and paper: error bars from a confusion matrix, the Wald
test, maximum likelihood for the binomial.
fundamentals-Uncertainty-lab.ipynb — programming: the coverage of a 95 % interval, the bootstrap,
comparing two classifiers paired, the maximum-likelihood estimate numerically.