Uncertainty

Machine Learning · Fundamentals

Error bars for estimates: standard errors · confidence intervals · maximum likelihood · the bootstrap · comparing two models

Why error bars?

Every number we report about a model is computed from a finite, random sample of data. A different test set would have given a different number. So

accuracy = 0.92 is not a result. accuracy = 0.92 ± 0.03 is.

This recap supplies the ±; the evaluation metrics carry it.

Prerequisite: the recaps Probability (random variables, expectation, variance, the binomial) and Estimators (estimator, bias, variance).

Learning objectives (1/2)

  1. compute a standard error and report a result as estimate ± uncertainty,
  2. construct and correctly interpret a 95 % confidence interval,
  3. derive the maximum-likelihood estimator $\hat\theta_{ML} = k/n$ of the binomial distribution, and recognise its negative log-likelihood as the cross-entropy,

Learning objectives (2/2)

  1. attach an error bar to the accuracy and to other rates of a classifier, such as precision and recall (defined in §4) — each with the right $n$,
  2. estimate a standard error for any metric with the bootstrap,
  3. decide whether two estimates really differ (Wald test, paired comparison).

Overview

Part The question it answers
1 · Standard error How large is the uncertainty of one estimate?
2 · Confidence intervals What exactly does "± 0.06" promise?
3 · Maximum likelihood Where does the estimator $k/n$ come from — and the cross-entropy?
4 · Proportions How large is the error bar of an accuracy — or of any other rate?
5 · The bootstrap And for a metric without a formula?
6 · Comparing two estimates Is model A really better than model B?
In [1]:
import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline
np.set_printoptions(precision=4, suppress=True)
plt.rcParams["figure.figsize"] = (9, 4)
plt.rcParams["axes.grid"] = True
plt.rcParams["grid.alpha"] = 0.3
rng = np.random.default_rng(0)

1. Standard error

The variance tells us how much $\hat\theta$ scatters over many data sets.

But we only ever have one data set. What can it tell us about its own uncertainty?

Definition

The standard error of an estimator is its standard deviation:

$$ \operatorname{se}(\hat\theta) = \sqrt{\operatorname{var}(\hat\theta)} $$

It has the same unit as $\theta$ — which is why we report it, not the variance.

For the sample mean (Estimators): $\;\operatorname{se}(\hat\mu) = \dfrac{\sigma}{\sqrt n}$.

The plug-in estimate

$\sigma$ is unknown — but it can itself be estimated from the same data:

$$ \widehat{\operatorname{se}}(\hat\mu) = \frac{\hat\sigma_u}{\sqrt n} $$

One data set gives both the estimate and its error bar.

That is the whole trick, and it works because the formula for $\operatorname{se}$ was derived before looking at the data.

In [2]:
mu_true, sigma_true = 3.0, 2.0                       # the Gaussian of Estimators §2
x = rng.normal(mu_true, sigma_true, size=25)          # ONE data set, n = 25
mu_hat_1 = x.mean()
se_hat_1 = x.std(ddof=1) / np.sqrt(len(x))            # sigma_u / sqrt(n)
print(f"mu = {mu_hat_1:.2f} ± {se_hat_1:.2f}   (true mu = {mu_true}, "
      f"true se = {sigma_true / 5:.2f})")
mu = 2.85 ± 0.35   (true mu = 3.0, true se = 0.40)

Does the error bar from one data set describe the scatter over many?

In [3]:
S25 = rng.normal(mu_true, sigma_true, size=(20_000, 25))   # 20 000 data sets
means = S25.mean(axis=1)
se_hats = S25.std(axis=1, ddof=1) / 5
print(f"std of the 20 000 estimates  : {means.std():.4f}")
print(f"theory sigma / sqrt(n)       : {sigma_true / 5:.4f}")
print(f"typical se_hat from ONE set  : {se_hats.mean():.4f}")
std of the 20 000 estimates  : 0.3997
theory sigma / sqrt(n)       : 0.4000
typical se_hat from ONE set  : 0.3964

Reporting a result

$$ \text{result} = (\text{best estimate} \pm \text{uncertainty})\;\text{unit} $$

  • round the uncertainty to one or two significant digits,
  • round the estimate to the same decimal place: $3.12 \pm 0.41$, not $3.1234 \pm 0.41$,
  • say what the ± means (next section) and how many data points it rests on.

Without an uncertainty, an estimate cannot be compared with anything — not with another model, not with last year's result.

Error versus uncertainty

  • the error $\hat\theta - \theta$ is the distance to the true value — unknown, because $\theta$ is unknown;
  • the uncertainty $\widehat{\operatorname{se}}$ quantifies how large that error typically is — computable from the data.

After S. Bell, A Beginner's Guide to Uncertainty of Measurement, NPL 2001.

2. Confidence intervals

The central limit theorem

For large $n$, a sample mean is approximately Gaussian — whatever the distribution of the individual $X_i$:

$$ \hat\mu \;\dot\sim\; \mathcal{N}\!\left(\mu,\; \operatorname{se}(\hat\mu)^2\right) $$

Many estimators behave the same way, in particular every proportion (§4).

Stated here, not proven (Wasserman 2004, Thm. 5.8); the figure below shows it.

In [4]:
fig, ax = plt.subplots(1, 3, figsize=(12, 3.2))
for a, n_ in zip(ax, [1, 5, 30]):
    m = rng.exponential(1.0, size=(50_000, n_)).mean(axis=1)   # skewed data
    a.hist(m, bins=80, density=True, color="steelblue")
    a.set_title(f"mean of n = {n_} exponential draws")
plt.tight_layout(); plt.show()
No description has been provided for this image

The 95 % confidence interval

95 % of a Gaussian lies within $1.96$ standard deviations of its mean (Probability). With $\hat\theta \;\dot\sim\; \mathcal N(\theta, \operatorname{se}^2)$:

$$ p\bigl(|\hat\theta - \theta| \le 1.96\,\operatorname{se}\bigr) \approx 0.95 $$

The event $|\hat\theta - \theta| \le 1.96\,\operatorname{se}$ is the same as $\hat\theta - 1.96\,\operatorname{se} \le \theta \le \hat\theta + 1.96\,\operatorname{se}$ — an interval around $\hat\theta$ that contains $\theta$ with probability $\approx 0.95$. In practice $\operatorname{se}$ is replaced by $\widehat{\operatorname{se}}$.

Careful. $\theta$ is fixed; the interval is random. "95 %" is a property of the procedure: of many intervals built this way, about 95 % contain $\theta$.

It is not "θ lies in this interval with probability 95 %".

Which ± ?

Both conventions are in use, so always say which one you mean:

written half-width coverage common in
$\hat\theta \pm \widehat{\operatorname{se}}$ 1 se ≈ 68 % physics, measurement
$\hat\theta \pm 1.96\,\widehat{\operatorname{se}}$ 1.96 se ≈ 95 % statistics, medicine

Convention in this course: $a \pm b$ means a 95 % confidence interval, $b = 1.96\,\widehat{\operatorname{se}}$ — unless stated otherwise.

In [5]:
def ci_covers(S, mu, z):
    '''For every row of S: does mean ± z * se_hat contain mu?'''
    m = S.mean(axis=1)
    se = S.std(axis=1, ddof=1) / np.sqrt(S.shape[1])
    return np.abs(m - mu) <= z * se
In [6]:
for z in [1.0, 1.96]:
    print(f"mean ± {z:4} se_hat contains mu in {ci_covers(S25, mu_true, z).mean():.1%}"
          " of 20 000 data sets")
mean ±  1.0 se_hat contains mu in 67.3% of 20 000 data sets
mean ± 1.96 se_hat contains mu in 94.0% of 20 000 data sets
In [7]:
cov = ci_covers(S25[:50], mu_true, 1.96)
lo = means[:50] - 1.96 * se_hats[:50]; hi = means[:50] + 1.96 * se_hats[:50]
fig, ax = plt.subplots(figsize=(11, 3.4))
for i in range(50):
    ax.plot([i, i], [lo[i], hi[i]], color="steelblue" if cov[i] else "red", lw=2)
ax.axhline(mu_true, color="k", lw=1.5, label=r"true $\mu$")
ax.set_xlabel("data set"); ax.set_title("50 data sets, 50 intervals (red: misses $\\mu$)")
ax.legend(); plt.show()
No description has been provided for this image

Small samples: the $t$ multiplier

$1.96$ assumes $\operatorname{se}$ is known. With $\widehat{\operatorname{se}}$ from very few points the interval must be wider: use the $t$-distribution with $n-1$ degrees of freedom (for Gaussian data).

$n$ 2 3 5 10 20 30 $\infty$
multiplier 12.71 4.30 2.78 2.26 2.09 2.05 1.96

This is why the simulation above, with $n = 25$ and $1.96$, covered slightly less than 95 %. Above $n \approx 30$ the difference is negligible — and test sets in ML are far larger.

3. Maximum likelihood for the binomial

So far the estimators were simply given: $\bar X$, $\hat\sigma_b^2$, $\hat\sigma_u^2$ (Estimators).

Where do estimators come from? The most important recipe is maximum likelihood. Logistic regression (→ L-LogReg) uses it to derive the cross-entropy; here is its simplest case — a proportion, such as the accuracy of a classifier.

The likelihood

A model $p(\mathcal D \mid \theta)$ gives the probability of the data for each value of the parameter. Read with the data fixed and $\theta$ varying, it is the likelihood:

$$ \mathcal L(\theta) = p(\mathcal D \mid \theta) $$

"Never say 'the likelihood of the data'. Always say 'the likelihood of the parameters'. The likelihood function is not a probability distribution."

D. MacKay, Information Theory, Inference and Learning Algorithms, 2003, p. 29

It need not integrate to 1 over $\theta$.

The maximum-likelihood estimator

Choose the parameter under which the observed data are most probable:

$$ \hat\theta_{ML} = \arg\max_\theta \mathcal L(\theta) = \arg\min_\theta \bigl(-\log \mathcal L(\theta)\bigr) $$

The logarithm is monotonic, so the location of the optimum does not move — but products become sums, which are easier to differentiate and numerically stable. $-\log\mathcal L$ is the negative log-likelihood (NLL).

Example: how often is a classifier right?

A classifier is applied to $n$ independent test examples. Each prediction is correct with the same unknown probability $\theta$. We observe $k$ correct predictions.

The number of correct predictions is binomially distributed (Probability):

$$ \mathcal L(\theta) = p(k \mid n, \theta) = \binom{n}{k}\,\theta^{k}(1-\theta)^{n-k} $$

In [8]:
def binom_nll(theta, k, n):
    '''Negative log-likelihood of the binomial, without the constant log C(n, k).'''
    return -(k * np.log(theta) + (n - k) * np.log(1 - theta))
In [9]:
th = np.linspace(0.4, 0.95, 500)
fig, ax = plt.subplots(1, 2, figsize=(11, 3.4))
for k_, n_ in [(7, 10), (70, 100)]:
    L = np.exp(-(binom_nll(th, k_, n_) - binom_nll(k_ / n_, k_, n_)))  # max scaled to 1
    ax[0].plot(th, L, label=f"k = {k_}, n = {n_}")
    ax[1].plot(th, binom_nll(th, k_, n_) / n_, label=f"k = {k_}, n = {n_}")
ax[0].set_title(r"likelihood $\mathcal{L}(\theta)$, scaled to max 1")
ax[1].set_title(r"mean NLL $-\frac{1}{n}\log\mathcal{L}(\theta)$")
for a in ax: a.axvline(0.7, color="k", ls=":"); a.set_xlabel(r"$\theta$"); a.legend()
plt.tight_layout(); plt.show()
No description has been provided for this image

Both curves peak at $\theta = 0.7$ — but the one built on ten times more data is much narrower. Keep that picture in mind: it will turn out to be the error bar.

Deriving $\hat\theta_{ML}$ (1/2)

Take the negative logarithm; the binomial coefficient does not depend on $\theta$:

$$ -\log\mathcal L(\theta) = -\log\binom{n}{k} \;-\; k\log\theta \;-\; (n-k)\log(1-\theta) $$

Set the derivative to zero — with $\frac{d}{d\theta}\log\theta = \frac1\theta$ and, by the chain rule, $\frac{d}{d\theta}\log(1-\theta) = -\frac{1}{1-\theta}$:

$$ 0 = \frac{d}{d\theta}\bigl(-\log\mathcal L\bigr) = -\frac{k}{\theta} + \frac{n-k}{1-\theta} $$

Deriving $\hat\theta_{ML}$ (2/2)

Multiply by $\theta(1-\theta)$:

$$ k(1-\theta) = (n-k)\,\theta \quad\Longrightarrow\quad k = n\theta \quad\Longrightarrow\quad \boxed{\;\hat\theta_{ML} = \frac{k}{n}\;} $$

Differentiating once more gives $\frac{k}{\theta^2} + \frac{n-k}{(1-\theta)^2} > 0$ for every $\theta \in (0, 1)$: the NLL is convex, so $k/n$ is its minimum.

The maximum-likelihood estimate of a probability is the observed frequency. An accuracy is an estimate of exactly this kind — and so is every other rate of a classifier that counts "$k$ out of $n$" (§4).

Check it numerically

Evaluate the NLL on a fine grid and look where the minimum is:

In [10]:
k_obs, n_obs = 88, 100
grid = np.linspace(0.001, 0.999, 99_801)
print(f"grid minimum of the NLL at theta = {grid[np.argmin(binom_nll(grid, k_obs, n_obs))]:.4f}")
print(f"k / n                           = {k_obs / n_obs:.4f}")
grid minimum of the NLL at theta = 0.8800
k / n                           = 0.8800

Relation to the cross-entropy

Write the data example by example: $y^{(i)} = 1$ if prediction $i$ is correct, else $0$, so $k = \sum_i y^{(i)}$. One example has $p(y^{(i)} \mid \theta) = \theta^{y^{(i)}}(1-\theta)^{1-y^{(i)}}$ — that is $\theta$ for $y = 1$ and $1 - \theta$ for $y = 0$. The examples are independent, so their probabilities multiply:

$$ \prod_{i=1}^n \theta^{y^{(i)}}(1-\theta)^{1-y^{(i)}} = \theta^{k}(1-\theta)^{n-k} $$

— the binomial likelihood without $\binom nk$, which only counts the orderings (a constant).

The log turns the product into a sum, so the mean NLL is

$$ -\frac{1}{n}\log\mathcal L(\theta) = -\frac{1}{n}\sum_{i=1}^{n}\Bigl[\,y^{(i)}\log\theta + \bigl(1-y^{(i)}\bigr)\log(1-\theta)\Bigr] \;(+\text{const}) $$

Compare the cost of logistic regression (→ L-LogReg):

$$ J(\vec\theta) = -\frac{1}{m}\sum_{i=1}^{m}\Bigl[\,y^{(i)}\log h_\theta(\vec x^{(i)}) + \bigl(1-y^{(i)}\bigr)\log\bigl(1-h_\theta(\vec x^{(i)})\bigr)\Bigr] $$

The same expression. Here the probability is one constant $\theta$ for every example; in logistic regression each example gets its own, $h_\theta(\vec x^{(i)}) = g(\vec\theta^{\,T}\vec x^{(i)})$, and the parameters are the weights inside $g$.

In [11]:
def cross_entropy(y, h):
    '''The binary cross-entropy of logistic regression, for labels y and predicted probabilities h.'''
    return -np.mean(y * np.log(h) + (1 - y) * np.log(1 - h))

y_obs = np.r_[np.ones(k_obs), np.zeros(n_obs - k_obs)]       # 88 correct, 12 wrong
for theta in [0.7, 0.88]:
    print(f"theta = {theta}:  cross-entropy = {cross_entropy(y_obs, theta):.5f}   "
          f"NLL / n = {binom_nll(theta, k_obs, n_obs) / n_obs:.5f}")
theta = 0.7:  cross-entropy = 0.45835   NLL / n = 0.45835
theta = 0.88:  cross-entropy = 0.36692   NLL / n = 0.36692

Minimising the cross-entropy is maximum-likelihood estimation — for a constant probability it returns $k/n$, for a logistic model it returns the weights $\vec\theta$.

The curvature is the error bar

Evaluate the second derivative of the NLL at its minimum $\hat\theta = k/n$, i.e. with $k = n\hat\theta$ and $n - k = n(1 - \hat\theta)$:

$$ \frac{d^2}{d\theta^2}\bigl(-\log\mathcal L\bigr)\Big|_{\hat\theta} = \frac{n\hat\theta}{\hat\theta^2} + \frac{n(1-\hat\theta)}{(1-\hat\theta)^2} = \frac{n}{\hat\theta} + \frac{n}{1-\hat\theta} = \frac{n}{\hat\theta(1-\hat\theta)} $$

Its inverse, $\hat\theta(1-\hat\theta)/n$, is exactly the variance of $\hat\theta$ that §4 derives directly.

This is no coincidence: in general, for large $n$,

$$ \widehat{\operatorname{se}}(\hat\theta_{ML}) \approx \Bigl(\tfrac{d^2}{d\theta^2}(-\log\mathcal L)\big|_{\hat\theta_{ML}}\Bigr)^{-1/2} $$

(the Fisher information; Wasserman 2004, §9.7 — stated here, not proven)

A sharper likelihood means a smaller error bar — the picture above.

In [12]:
eps = 1e-4                                    # finite-difference step
t0 = k_obs / n_obs
curv = (binom_nll(t0 + eps, k_obs, n_obs) - 2 * binom_nll(t0, k_obs, n_obs)
        + binom_nll(t0 - eps, k_obs, n_obs)) / eps**2
print(f"se from the curvature : {1 / np.sqrt(curv):.4f}")
print(f"sqrt(theta(1-theta)/n): {np.sqrt(t0 * (1 - t0) / n_obs):.4f}")
se from the curvature : 0.0325
sqrt(theta(1-theta)/n): 0.0325

4. Proportions — error bars for evaluation metrics

A proportion is a sample mean

Let $X_i \in \{0, 1\}$ with $p(X_i = 1) = \theta$ — "prediction $i$ is correct". Then

$$ \hat\theta = \frac{k}{n} = \frac{1}{n}\sum_{i=1}^{n} X_i $$

is a sample mean — so everything from §1 applies, and so does Estimators.

Mean, variance, standard error

Since $X_i^2 = X_i$: $\;\mathbb{E}[X_i] = \theta$, $\;\operatorname{var}(X_i) = \mathbb{E}[X_i^2] - \theta^2 = \theta(1-\theta)$ (Probability).

With $\operatorname{var}(\bar X) = \sigma^2/n$ from Estimators:

$$ \mathbb{E}[\hat\theta] = \theta, \qquad \operatorname{var}(\hat\theta) = \frac{\theta(1-\theta)}{n}, \qquad \widehat{\operatorname{se}}(\hat\theta) = \sqrt{\frac{\hat\theta(1-\hat\theta)}{n}} $$

The error bar of a proportion needs no extra data: $\hat\theta$ and $n$ are enough.

Its size: $\theta(1-\theta)$ is largest at $\theta = 0.5$ (Probability), so at worst $\pm 1.96\sqrt{0.5 \cdot 0.5/n} \approx \pm 1/\sqrt n$ — about $\pm 0.10$ for $n = 100$, $\pm 0.03$ for $n = 1000$.

More rates than accuracy

For a binary classifier, every test example falls into one of four groups — the confusion matrix:

predicted 1 predicted 0
actual 1 true positive (TP) false negative (FN)
actual 0 false positive (FP) true negative (TN)

Accuracy is $(\text{TP} + \text{TN})/n$. It hides which mistakes are made.

Two further rates, each a conditional probability (Probability):

  • precision $= \dfrac{\text{TP}}{\text{TP} + \text{FP}} = p(y = 1 \mid \hat y = 1)$ — of the examples predicted positive, the share that really is positive;
  • recall $= \dfrac{\text{TP}}{\text{TP} + \text{FN}} = p(\hat y = 1 \mid y = 1)$ — of the actual positives, the share the classifier finds.

Why these are needed, and further rates, are the subject of → L-Eval. Here they only need error bars.

Example: a confusion matrix

A classifier evaluated on $400$ test examples:

predicted 1 predicted 0
actual 1 TP = 88 FN = 22
actual 0 FP = 12 TN = 278

Each metric has its own $n$

metric $k$ $n$ = the examples it is computed on
accuracy TP + TN all examples, 400
precision TP predicted positives, TP + FP = 100
recall TP actual positives, TP + FN = 110
In [13]:
def proportion_ci(k, n, z=1.96):
    '''Estimate k/n and the half-width z * se_hat of its confidence interval.'''
    theta_hat = k / n
    return theta_hat, z * np.sqrt(theta_hat * (1 - theta_hat) / n)
In [14]:
TP, FN, FP, TN = 88, 22, 12, 278
for name, k_, n_ in [("accuracy", TP + TN, TP + FN + FP + TN),
                     ("precision", TP, TP + FP), ("recall", TP, TP + FN)]:
    est, half = proportion_ci(k_, n_)
    print(f"{name:9} = {est:.2f} ± {half:.2f}   (95 % CI, n = {n_})")
accuracy  = 0.92 ± 0.03   (95 % CI, n = 400)
precision = 0.88 ± 0.06   (95 % CI, n = 100)
recall    = 0.80 ± 0.07   (95 % CI, n = 110)

The common mistake: computing the precision's error bar with $n = 400$. That gives $\pm 0.03$ — too narrow by the factor $\sqrt{400/100} = 2$. Only the 100 examples predicted positive say anything about precision; the other 300 carry no information about it.

When the normal approximation fails

$\hat\theta \pm 1.96\,\widehat{\operatorname{se}}$ relies on the central limit theorem. It breaks down when $n$ is small or $\hat\theta$ is near 0 or 1: with $10$ correct out of $10$, $\widehat{\operatorname{se}} = 0$ and the "interval" is the single point $1$.

Rule of thumb: use it only if $n\hat\theta \ge 5$ and $n(1-\hat\theta) \ge 5$. Otherwise use the Wilson interval below, or the bootstrap of §5 with care.

The Wilson score interval (with $z = 1.96$) has centre and half-width

$$ \frac{\hat\theta + \frac{z^2}{2n}}{1 + \frac{z^2}{n}} \qquad\text{and}\qquad \frac{z}{1 + \frac{z^2}{n}}\sqrt{\frac{\hat\theta(1-\hat\theta)}{n} + \frac{z^2}{4n^2}} $$

For 10 out of 10 it gives $[0.72,\, 1.00]$ — an honest statement that ten trials are not many.

5. The bootstrap

Precision and recall have a formula for their error bar. Many metrics do not — for example the $F_1$ score, which combines precision and recall into one number (their harmonic mean; the reason is given in → L-Eval):

$$ F_1 = \frac{2\cdot\text{precision}\cdot\text{recall}}{\text{precision}+\text{recall}} $$

— or the median. Ideally we would draw many new test sets and look at the spread of the metric — the simulations of Estimators. We cannot: we have one test set.

The plug-in idea

Replace the unknown distribution of the data by the empirical distribution of the $n$ points we have — and simulate from that.

Drawing from the empirical distribution = drawing $n$ points from our data set with replacement. Each such resample is a plausible "other test set".

B. Efron, 1979. Wasserman 2004, ch. 8.

The algorithm

For $b = 1, \dots, B$ (typically $B = 1000$ to $10\,000$):

  1. draw $n$ indices uniformly with replacement from $\{1, \dots, n\}$,
  2. compute the statistic on these rows: $\hat\theta^*_b$.

Then

  • $\widehat{\operatorname{se}}_{boot} = \operatorname{std}(\hat\theta^*_1, \dots, \hat\theta^*_B)$,
  • percentile interval: the 2.5 % and 97.5 % quantiles of the $\hat\theta^*_b$.
In [15]:
def bootstrap(stat, data, B=5000, rng=rng):
    '''B bootstrap replicates of stat(data); data has one example per row.'''
    n = len(data)
    reps = np.empty(B)
    for b in range(B):
        idx = rng.integers(0, n, size=n)       # n draws WITH replacement
        reps[b] = stat(data[idx])
    return reps

Sanity check on a case with a formula — the precision of §4: 88 ones, 12 zeros.

In [16]:
correct = np.r_[np.ones(88), np.zeros(12)]
reps = bootstrap(np.mean, correct)
print(f"bootstrap se : {reps.std():.4f}")
print(f"formula se   : {np.sqrt(0.88 * 0.12 / 100):.4f}")
bootstrap se : 0.0324
formula se   : 0.0325

$F_1$ with an error bar

Resample what was sampled: whole test examples, i.e. pairs $(y, \hat y)$ — not the four cells of the confusion matrix independently.

In [17]:
def f1(pairs):
    '''F1 from an array of (y, y_hat) rows.'''
    y, yh = pairs[:, 0], pairs[:, 1]
    tp = np.sum((y == 1) & (yh == 1))
    return 2 * tp / (2 * tp + np.sum((y == 0) & (yh == 1)) + np.sum((y == 1) & (yh == 0)))

pairs = np.array([[1, 1]] * TP + [[1, 0]] * FN + [[0, 1]] * FP + [[0, 0]] * TN)
In [18]:
reps_f1 = bootstrap(f1, pairs)
lo, hi = np.quantile(reps_f1, [0.025, 0.975])
print(f"F1 = {f1(pairs):.2f} ± {1.96 * reps_f1.std():.2f}   "
      f"(95 % CI from 1.96 se_boot)")
print(f"percentile interval: [{lo:.2f}, {hi:.2f}]")
F1 = 0.84 ± 0.05   (95 % CI from 1.96 se_boot)
percentile interval: [0.78, 0.89]
In [19]:
fig, ax = plt.subplots(figsize=(8, 3))
ax.hist(reps_f1, bins=60, color="steelblue")
ax.axvline(f1(pairs), color="k", lw=2, label="F1 on the test set")
for q in (lo, hi): ax.axvline(q, color="red", ls="--")
ax.set_title("5000 bootstrap replicates of $F_1$"); ax.legend(); plt.show()
No description has been provided for this image

The bootstrap estimates the variance due to the finite test set. It cannot repair a test set that is too small, unrepresentative, or has been used to choose the model — those are bias problems.

6. Comparing two estimates

Model A: precision $0.88 \pm 0.06$. Model B: precision $0.81 \pm 0.07$.

Is A better — or is the difference just test-set noise?

Reading overlapping error bars

For two independent estimates with 95 % bars of similar size:

  • the bars do not touch → the difference is significant (in fact at well below 5 %);
  • the bars overlap by up to about half an arm → still significant at roughly 5 %;
  • the bars overlap heavily → no evidence of a difference.

Overlap means "not shown", never "shown to be equal".

The Wald test

For independent estimates the covariance vanishes and the variances of the difference add (Probability): $\operatorname{var}(\hat\theta_A - \hat\theta_B) = \operatorname{se}_A^2 + \operatorname{se}_B^2$. Dividing the difference by its standard error gives variance 1 (Rule 1), and by the CLT it is near-Gaussian. If the true values are equal,

$$ Z = \frac{\hat\theta_A - \hat\theta_B}{\sqrt{\widehat{\operatorname{se}}_A^2 + \widehat{\operatorname{se}}_B^2}} \;\dot\sim\; \mathcal N(0, 1) $$

so $|Z| > 1.96$ happens by chance only 5 % of the time. Wasserman 2004, §10.1.

This is where the overlap rules come from. Take equal standard errors, so the difference has standard error $\sqrt2\,\operatorname{se}$:

  • the 95 % bars stop touching at a gap of $2 \cdot 1.96\,\operatorname{se} = 3.92\,\operatorname{se}$, i.e. $Z = 3.92/\sqrt2 = 2.77$ — by chance only 0.6 % of the time;
  • overlapping by half an arm, the gap is $2.94\,\operatorname{se}$, $Z = 2.08$ — about 4 %;
  • the Wald test needs only $1.96\sqrt2\,\operatorname{se} \approx 2.8\,\operatorname{se}$.

Bars may overlap a little while the difference is significant.

In [20]:
n_sim, sig = 10, 1.0
se = sig / np.sqrt(n_sim)
A = rng.normal(0.0, sig, (100_000, n_sim)).mean(axis=1)   # two estimates of the
B = rng.normal(0.0, sig, (100_000, n_sim)).mean(axis=1)   # SAME true value 0
gap = np.abs(A - B) / se
print(f"Wald |Z| > 1.96            : {np.mean(gap / np.sqrt(2) > 1.96):.3f}  (target 0.05)")
print(f"95 % bars do not touch     : {np.mean(gap > 2 * 1.96):.3f}")
print(f"overlap by half an arm     : {np.mean(gap > 1.5 * 1.96):.3f}")
Wald |Z| > 1.96            : 0.050  (target 0.05)
95 % bars do not touch     : 0.005
overlap by half an arm     : 0.038

Paired comparison

Two models evaluated on the same test set are not independent: both tend to get the easy examples right and the hard ones wrong.

Use the per-example difference of the correctness indicators, $d^{(i)} = c_A^{(i)} - c_B^{(i)} \in \{-1, 0, 1\}$:

$$ \hat\theta_A - \hat\theta_B = \bar d, \qquad \widehat{\operatorname{se}}(\bar d) = \frac{\operatorname{std}(d)}{\sqrt n} $$

$\bar d$ is a sample mean, so its standard error is that of §4. It is small because $\operatorname{var}(d) = \operatorname{var}(c_A) + \operatorname{var}(c_B) - 2\operatorname{Cov}(c_A, c_B)$ (Probability), and the covariance is positive: examples both models get right (or both wrong) contribute $d = 0$ — they cancel.

In [21]:
def two_classifiers(n, acc_a, acc_b, rng, redraw=0.2):
    '''Correctness of A and B on the same n examples, sharing example difficulty.'''
    u = rng.random(n)                                         # difficulty
    u_b = np.where(rng.random(n) < redraw, rng.random(n), u)  # B differs on 20 %
    return (u < acc_a).astype(float), (u_b < acc_b).astype(float)
In [22]:
def z_independent(ca, cb):
    a, b, n = ca.mean(), cb.mean(), len(ca)
    return (a - b) / np.sqrt(a * (1 - a) / n + b * (1 - b) / n)

def z_paired(ca, cb):
    d = ca - cb
    return d.mean() / (d.std(ddof=1) / np.sqrt(len(d)))
In [23]:
sim_rng = np.random.default_rng(1)
z_i, z_p = [], []
for _ in range(2000):                                  # 2000 test sets, n = 500
    ca, cb = two_classifiers(500, 0.86, 0.82, sim_rng)
    z_i.append(z_independent(ca, cb)); z_p.append(z_paired(ca, cb))
print("A is truly 4 points better. Share of test sets where the test detects it:")
print(f"  independent Wald : {np.mean(np.array(z_i) > 1.96):.2f}")
print(f"  paired           : {np.mean(np.array(z_p) > 1.96):.2f}")
A is truly 4 points better. Share of test sets where the test detects it:
  independent Wald : 0.35
  paired           : 0.89

On a shared test set, always compare paired. The independent test throws away the fact that both models saw the same examples, and misses real differences.

The bootstrap version: resample example indices once, compute both metrics on that resample, and look at the spread of their difference (a paired bootstrap), as used in → L-Eval.

7. Summary

Take away (1/2)

  1. The standard error $\operatorname{se} = \sqrt{\operatorname{var}}$ can be estimated from the one data set we have; for a mean it shrinks like $1/\sqrt n$.
  2. Report estimate ± uncertainty and say what ± means — here: a 95 % CI, $\pm 1.96\,\widehat{\operatorname{se}}$.
  3. Maximum likelihood gives $\hat\theta_{ML} = k/n$ for the binomial; its NLL is the cross-entropy, and its curvature gives the error bar.

Take away (2/2)

  1. Every rate metric is a proportion: $\widehat{\operatorname{se}} = \sqrt{\hat\theta(1-\hat\theta)/n}$ with its own $n$ (precision: the predicted positives).
  2. No formula? Bootstrap — resample whole test examples.
  3. Same test set, two models? Compare paired.

Exercises and Lab

fundamentals-Uncertainty-exercises.ipynb — pen and paper: error bars from a confusion matrix, the Wald test, maximum likelihood for the binomial.

fundamentals-Uncertainty-lab.ipynb — programming: the coverage of a 95 % interval, the bootstrap, comparing two classifiers paired, the maximum-likelihood estimate numerically.

Literature

  • L. Wasserman: All of Statistics, Springer 2004 — ch. 6 (estimators, confidence intervals), ch. 8 (bootstrap), §9.7 (Fisher information), §10.1 (Wald test). doi:10.1007/978-0-387-21736-9
  • I. Goodfellow, Y. Bengio, A. Courville: Deep Learning, MIT Press 2016 — §5.4 (estimators, bias, variance), §5.5 (maximum likelihood).
  • D. MacKay: Information Theory, Inference and Learning Algorithms, Cambridge 2003, ch. 3.
  • B. Efron, R. Tibshirani: An Introduction to the Bootstrap, Chapman & Hall 1993.
  • S. Bell: A Beginner's Guide to Uncertainty of Measurement, NPL 2001.