Probability

Machine Learning · Fundamentals

Events · random variables · conditional probability · Bayes' rule · expectation and variance · the binomial and the Gaussian

Why probability?

  • Logistic regression (→ L-LogReg) already is a probability model: $h_\theta(\vec x) = p(y = 1 \mid \vec x)$.
  • Every number we report about a model — an accuracy, an error rate — is computed from a random test set.

Learning objectives (1/2)

  1. use sample space, events and the axioms of probability,
  2. define a random variable and its distribution, including the Bernoulli distribution,
  3. compute joint, marginal and conditional probabilities from a table,
  4. decide whether two variables are independent or conditionally independent,
  5. apply Bayes' rule and explain the base-rate fallacy,

Learning objectives (2/2)

  1. compute expectation, variance and covariance, and use their rules — linearity, $\operatorname{var}(aX+b) = a^2\operatorname{var}(X)$, the variance of a sum,
  2. derive mean and variance of the binomial distribution,
  3. distinguish a density from a probability, and use the Gaussian,
  4. explain why a simulation can check a probability — the law of large numbers.

Overview

Part The question it answers
1 · Outcome, event, sample space Which rules does every probability obey?
2 · Random variables How is a number attached to a random result?
3 · The law of large numbers Why does counting in a simulation estimate a probability?
4 · Joint and marginal How do two random variables behave together, and each alone?
5 · Conditional probability and independence How does knowing one value change the other?
6 · Bayes' rule How do we turn $p(A \mid B)$ into $p(B \mid A)$?
7 · Conditional independence When does a third variable make two variables independent?
8 · Expectation · 9 · Variance and covariance What is the average value, and how much does it scatter?
10 · The binomial distribution How many successes in $n$ trials?
11 · Continuous variables and the Gaussian What replaces probabilities when the values are real numbers?
In [1]:
import math
import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline
np.set_printoptions(precision=4, suppress=True)
plt.rcParams["figure.figsize"] = (9, 4)
plt.rcParams["axes.grid"] = True
plt.rcParams["grid.alpha"] = 0.3
rng = np.random.default_rng(0)

Method: derivation first, simulation second. Every rule is derived from the axioms; a simulation then checks it by counting.

Notation: $p(\cdot)$ — lower case — for every probability, probability mass and density. Many books write $P$.

1. Outcome, event, sample space

  • Outcome (Ergebnis): one possible result of a random experiment — a die shows $3$.
  • Sample space $\Omega$: the set of all outcomes, $\Omega = \{1, 2, 3, 4, 5, 6\}$.
  • Event (Ereignis): a subset of $\Omega$ — "even" $= \{2, 4, 6\}$, "greater than 4" $= \{5, 6\}$.

The three axioms

A probability $p$ assigns a number to every event $A$ such that

  1. $p(A) \ge 0$,
  2. $p(\Omega) = 1$,
  3. for disjoint events $A_1, A_2, \dots$ (no shared outcome): $p(A_1 \cup A_2 \cup \dots) = p(A_1) + p(A_2) + \dots$

Wasserman 2004, Def. 1.5. Everything else in this recap is derived from these three.

Two consequences

An event's probability is the sum over its outcomes — the single outcomes are disjoint, so axiom 3 applies: $\;p(A) = \sum_{\omega \in A} p(\{\omega\})$.

Fair die: six equal $p(\{\omega\}) = q$ with $6q = p(\Omega) = 1$, so $q = 1/6$ and $p(\text{even}) = 3/6$.

The complement $A^c = \Omega \setminus A$ is disjoint from $A$, and $A \cup A^c = \Omega$:

$$ p(A^c) = 1 - p(A) $$

Often the easy way round: $p(\text{at least one six in two rolls}) = 1 - p(\text{no six}) = 1 - (5/6)^2$. Each roll shows no six with probability $5/6$; the two probabilities multiply because the rolls are independent (§5).

In [2]:
rolls = rng.integers(1, 7, size=10_000)          # 10 000 rolls of a fair die
faces = np.arange(1, 7)
freq = np.array([np.mean(rolls == f) for f in faces])
print("relative frequency of each face:", freq.round(3), "  (1/6 = 0.167)")
print(f"relative frequency of 'even'   : {np.mean(rolls % 2 == 0):.3f}   (3/6 = 0.5)")
relative frequency of each face: [0.169 0.164 0.164 0.169 0.165 0.169]   (1/6 = 0.167)
relative frequency of 'even'   : 0.502   (3/6 = 0.5)

2. Random variables

A random variable is a function $X: \Omega \to \mathbb R$ that attaches a number to every outcome. Capital letters for the variable, lower case for its values.

Its set of possible values is $\operatorname{Val}(X)$.

Die: $X = 1$ if the face is odd, else $0$; $\;\operatorname{Val}(X) = \{0, 1\}$.

A value is an event

$\{X = x\}$ is short for the event $\{\omega \in \Omega : X(\omega) = x\}$, so

$$ p(X{=}1) = p(\{1, 3, 5\}) = \tfrac12 . $$

Notation: $p(X)$ is the whole distribution — a table or a function; $p(X{=}x)$ is one number; $p(x)$ is short for $p(X{=}x)$ when unambiguous.

Discrete distributions

A discrete distribution assigns a probability to each value:

$$ p(X{=}x) \ge 0 \qquad\text{and}\qquad \sum_{x \in \operatorname{Val}(X)} p(X{=}x) = 1 $$

Both follow from the axioms: the events $\{X = x\}$ for different $x$ are disjoint and together form $\Omega$.

The Bernoulli distribution

Two values, $1$ ("success") and $0$ ("failure"):

$$ p(X{=}1) = \theta, \qquad p(X{=}0) = 1 - \theta, \qquad X \sim \mathrm{Bernoulli}(\theta) $$

  • the die variable "odd" is $\mathrm{Bernoulli}(0.5)$;
  • "the classifier is right on test example $i$" is $\mathrm{Bernoulli}(\theta)$, with $\theta$ the accuracy we want to know;
  • logistic regression (→ L-LogReg): $y \mid \vec x \sim \mathrm{Bernoulli}\bigl(h_\theta(\vec x)\bigr)$.

3. The law of large numbers

Repeat an experiment $n$ times independently and count how often event $A$ occurs. As $n$ grows, the relative frequency approaches $p(A)$.

More generally: the mean of $n$ independent repetitions of a random variable approaches its expected value (§8). Wasserman 2004, Thm. 5.6.

This is why every simulation in this course can check a probability by counting.

In [3]:
even = (rolls % 2 == 0)
running = np.cumsum(even) / np.arange(1, len(rolls) + 1)

fig, ax = plt.subplots(figsize=(9, 3.2))
ax.plot(np.arange(1, len(rolls) + 1), running, lw=1.5)
ax.axhline(0.5, color="k", ls=":", label=r"$p(\mathrm{even}) = 0.5$")
ax.set_xscale("log"); ax.set_xlabel("number of rolls n"); ax.set_ylim(0, 1)
ax.set_title("running relative frequency of 'even'"); ax.legend(); plt.show()
No description has been provided for this image

The frequency settles — but slowly: its typical distance from $0.5$ shrinks like $1/\sqrt n$ — §9 shows why, and the recap Uncertainty turns it into an error bar.

4. Joint and marginal

Two variables on one die

$X = 1$ if the face is odd, $Y = \mathbb 1[\text{face} > 3]$ — the indicator: $1$ if the condition holds, else $0$.

The joint distribution $p(X, Y)$ gives every combination:

$p(X, Y)$ $Y = 0$ (faces 1–3) $Y = 1$ (faces 4–6)
$X = 1$ (odd) $2/6$ — faces 1, 3 $1/6$ — face 5
$X = 0$ (even) $1/6$ — face 2 $2/6$ — faces 4, 6

Marginalisation

The distribution of one variable: sum the joint over the other.

$$ p(X{=}x) = \sum_{y} p(X{=}x, Y{=}y) $$

The events $\{X{=}x, Y{=}y\}$ for different $y$ are disjoint and together give $\{X = x\}$.

Die: $p(X{=}1) = 2/6 + 1/6 = 1/2$ — the row sum. The column sums give $p(Y)$.

In [4]:
X = (rolls % 2 == 1).astype(int)                 # odd
Y = (rolls > 3).astype(int)                      # greater than 3
joint = np.array([[np.mean((X == x) & (Y == y)) for y in (0, 1)] for x in (0, 1)])
print("estimated p(X=x, Y=y)  (rows x = 0, 1; columns y = 0, 1)")
print(joint.round(3), "   exact: [[1/6, 2/6], [2/6, 1/6]]")
print("marginal p(X):", joint.sum(axis=1).round(3), "   marginal p(Y):", joint.sum(axis=0).round(3))
estimated p(X=x, Y=y)  (rows x = 0, 1; columns y = 0, 1)
[[0.164 0.338]
 [0.333 0.165]]    exact: [[1/6, 2/6], [2/6, 1/6]]
marginal p(X): [0.502 0.498]    marginal p(Y): [0.497 0.503]

5. Conditional probability and independence

Conditional probability

$$ p(X \mid Y) = \frac{p(X, Y)}{p(Y)}, \qquad p(Y) > 0 $$

"Probability of $X$ given $Y$". Conditioning restricts the sample space to the outcomes with $Y = y$ and rescales them to sum to 1.

Die: $p(X{=}1 \mid Y{=}1) = \dfrac{1/6}{1/2} = \dfrac13$ — of the faces 4, 5, 6 only 5 is odd.

In [5]:
kept = rolls[Y == 1]                              # keep only the rolls with face > 3
print(f"faces kept: {np.unique(kept)}")
print(f"share of odd faces among them: {np.mean(kept % 2 == 1):.3f}   (1/3 = 0.333)")
faces kept: [4 5 6]
share of odd faces among them: 0.328   (1/3 = 0.333)

The product rule

Multiply the definition by $p(Y)$:

$$ p(X, Y) = p(X \mid Y)\,p(Y) = p(Y \mid X)\,p(X) $$

Both ways round — the second line of §6 starts here.

Independence

$X$ and $Y$ are independent, $X \perp Y$, if for all values

$$ p(X, Y) = p(X)\,p(Y) $$

Equivalently, knowing $Y$ does not change the distribution of $X$:

$$ p(X \mid Y) = \frac{p(X, Y)}{p(Y)} = \frac{p(X)\,p(Y)}{p(Y)} = p(X) $$

One die: $p(X{=}1, Y{=}1) = 1/6 \ne p(X{=}1)\,p(Y{=}1) = 1/4$ — dependent: knowing "greater than 3" makes "odd" less likely.

Two fair dice, $X$ from the first and $Y$ from the second: all $36$ pairs of faces are equally likely, so $p(i, j) = \frac{1}{36} = \frac16 \cdot \frac16 = p(i)\,p(j)$ — independent, and so are $X$ and $Y$.

In [6]:
second = rng.integers(1, 7, size=len(rolls))     # a second, separate die
Y2 = (second > 3).astype(int)
print(f"one die : p(X=1, Y=1) = {np.mean((X == 1) & (Y == 1)):.3f}   "
      f"p(X=1) p(Y=1) = {X.mean() * Y.mean():.3f}")
print(f"two dice: p(X=1, Y=1) = {np.mean((X == 1) & (Y2 == 1)):.3f}   "
      f"p(X=1) p(Y=1) = {X.mean() * Y2.mean():.3f}")
one die : p(X=1, Y=1) = 0.165   p(X=1) p(Y=1) = 0.251
two dice: p(X=1, Y=1) = 0.255   p(X=1) p(Y=1) = 0.248

6. Bayes' rule

$$ \begin{aligned} p(B \mid A) &= \frac{p(A, B)}{p(A)} && \text{definition} \\ &= \frac{p(A \mid B)\,p(B)}{p(A)} && \text{product rule} \end{aligned} $$

and the denominator by marginalisation (§4) and the product rule:

$$ p(A) = \sum_{b} p(A, B{=}b) = \sum_{b} p(A \mid B{=}b)\,p(B{=}b) $$

Example: a screening test

  • 1 % of the population is ill: $p(\text{ill}) = 0.01$ — the prevalence or base rate,
  • the test detects 90 % of the ill: $p(\text{pos} \mid \text{ill}) = 0.9$,
  • it raises a false alarm for 5 % of the healthy: $p(\text{pos} \mid \text{healthy}) = 0.05$.

A patient tests positive. How likely is the disease, $p(\text{ill} \mid \text{pos})$?

$$ p(\text{ill} \mid \text{pos}) = \frac{0.9 \cdot 0.01}{0.9 \cdot 0.01 + 0.05 \cdot 0.99} = \frac{0.009}{0.0585} \approx 0.15 $$

A "90 % test" — and 85 % of the positive tests belong to healthy people. The healthy are so many that their 5 % false alarms outnumber the true cases: the base-rate fallacy.

In [7]:
n_people = 1_000_000
ill = rng.random(n_people) < 0.01
pos = np.where(ill, rng.random(n_people) < 0.9, rng.random(n_people) < 0.05)
print(f"share of the positive tests that are ill: {ill[pos].mean():.3f}")
print(f"Bayes' rule                             : {0.009 / (0.009 + 0.05 * 0.99):.3f}")
share of the positive tests that are ill: 0.154
Bayes' rule                             : 0.154

$p(\text{pos} \mid \text{ill})$ and $p(\text{pos} \mid \text{healthy})$ are properties of the test; $p(\text{ill} \mid \text{pos})$ depends on the base rate as well.

A classifier is a test of exactly this kind. The evaluation metrics give these conditional probabilities their names (→ L-Eval), and the fairness metrics compare base rates between groups (→ L-Fair).

7. Conditional independence

$X$ and $Y$ are conditionally independent given $Z$, $X \perp Y \mid Z$, if

$$ p(X, Y \mid Z) = p(X \mid Z)\,p(Y \mid Z) \qquad\Longleftrightarrow\qquad p(X \mid Y, Z) = p(X \mid Z) $$

Once $Z$ is known, $Y$ carries no further information about $X$.

Why the two forms are equivalent.

$$ \begin{aligned} p(X \mid Y, Z) &= \frac{p(X, Y \mid Z)}{p(Y \mid Z)} && \text{definition, everything given } Z \\ &= \frac{p(X \mid Z)\,p(Y \mid Z)}{p(Y \mid Z)} && \text{first form} \\ &= p(X \mid Z) \end{aligned} $$

Read backwards, with the product rule, the second form gives the first.

Dependent, yet independent given $Z$

Pick a coin at random: fair ($Z = 0$, heads $0.5$) or biased ($Z = 1$, heads $0.9$). Toss it twice; $X$, $Y$ = heads on toss 1, 2. Given the coin, the tosses are independent.

$$ \begin{aligned} p(X{=}1) &= \textstyle\sum_z p(z)\,p(X{=}1 \mid z) = \tfrac12 \cdot 0.5 + \tfrac12 \cdot 0.9 = 0.7 && \text{marginalise over } Z \\ p(X{=}1, Y{=}1) &= \textstyle\sum_z p(z)\,p(X{=}1 \mid z)\,p(Y{=}1 \mid z) = \tfrac12 \cdot 0.5^2 + \tfrac12 \cdot 0.9^2 = 0.53 && \text{and } X \perp Y \mid Z \end{aligned} $$

$0.53 \ne 0.7 \cdot 0.7$: without the coin, the tosses are dependent.

A first head hints at the biased coin, and so at a second head.

In [8]:
reps = 100_000
Z = rng.integers(0, 2, size=reps)                 # which coin
heads = np.where(Z == 1, 0.9, 0.5)
Xc = (rng.random(reps) < heads).astype(int)       # toss 1
Yc = (rng.random(reps) < heads).astype(int)       # toss 2, same coin
print(f"without Z : p(X=1,Y=1) = {np.mean(Xc & Yc):.3f}   p(X=1)p(Y=1) = {Xc.mean() * Yc.mean():.3f}")
for z in (0, 1):
    m = Z == z
    print(f"given Z={z} : p(X=1,Y=1|Z) = {np.mean(Xc[m] & Yc[m]):.3f}   "
          f"p(X=1|Z)p(Y=1|Z) = {Xc[m].mean() * Yc[m].mean():.3f}")
without Z : p(X=1,Y=1) = 0.534   p(X=1)p(Y=1) = 0.495
given Z=0 : p(X=1,Y=1|Z) = 0.255   p(X=1|Z)p(Y=1|Z) = 0.254
given Z=1 : p(X=1,Y=1|Z) = 0.810   p(X=1|Z)p(Y=1|Z) = 0.810

Independence and conditional independence do not imply each other.

8. Expectation

The expected value is the probability-weighted average of the values:

$$ \mathbb{E}[X] = \sum_{x \in \operatorname{Val}(X)} x\, p(X{=}x) $$

Bernoulli: $\mathbb E[X] = 1\cdot\theta + 0\cdot(1-\theta) = \theta$. Fair die: $\mathbb E[\text{face}] = \frac16(1 + \dots + 6) = 3.5$ — not a possible face.

The expectation of a function

Often we need the average of a function of $X$ — a square, a loss, a logarithm. $g(X)$ is itself a random variable, and its expectation needs no new distribution:

$$ \mathbb{E}[g(X)] = \sum_{x \in \operatorname{Val}(X)} g(x)\, p(X{=}x) $$

Weight $g$ of each value with the probability of that value. For two variables, likewise $\;\mathbb{E}[g(X, Y)] = \sum_{x, y} g(x, y)\, p(x, y)$.

Die: $\mathbb E[F^2] = \frac16(1 + 4 + 9 + 16 + 25 + 36) = \frac{91}{6} \approx 15.17$.

In general $\mathbb E[g(X)] \ne g(\mathbb E[X])$: here $(\mathbb E F)^2 = 3.5^2 = 12.25$. The gap, $15.17 - 12.25$, is exactly the variance of §9.

Everything that follows is of this form: the variance $\mathbb E[(X - \mu)^2]$, the covariance $\mathbb E[(X - \mu_X)(Y - \mu_Y)]$, and the mean squared error $\mathbb E[(\hat\theta - \theta)^2]$.

In [9]:
F = rng.integers(1, 7, size=100_000)
print(f"mean of F^2 over 100 000 rolls: {np.mean(F**2):.3f}   (E[F^2] = 91/6 = {91 / 6:.3f})")
print(f"(mean of F)^2                 : {F.mean()**2:.3f}   ((E F)^2 = 12.25)")
mean of F^2 over 100 000 rolls: 15.188   (E[F^2] = 91/6 = 15.167)
(mean of F)^2                 : 12.265   ((E F)^2 = 12.25)

Linearity

For random variables $X, Y$ and constants $a, b$ — dependent or not:

$$ \begin{aligned} \mathbb{E}[aX + bY] &= \textstyle\sum_{x, y} (ax + by)\, p(x, y) && \mathbb E[g(X, Y)] \\ &= a \textstyle\sum_x x \sum_y p(x, y) + b \sum_y y \sum_x p(x, y) && \text{split the sum} \\ &= a\,\mathbb{E}[X] + b\,\mathbb{E}[Y] && \text{inner sums are marginals (§4)} \end{aligned} $$

Consequence: the sample mean $\bar X = \frac1n\sum_i X_i$ of variables with $\mathbb E[X_i] = \mu$ has $\mathbb E[\bar X] = \mu$. The law of large numbers (§3) says more: for independent $X_i$, $\bar X$ itself approaches $\mu$.

In [10]:
for n in [10, 1_000, 100_000, 1_000_000]:
    print(f"mean of {n:6} die rolls: {rng.integers(1, 7, size=n).mean():.3f}   (E = 3.5)")
mean of     10 die rolls: 2.800   (E = 3.5)
mean of   1000 die rolls: 3.493   (E = 3.5)
mean of 100000 die rolls: 3.492   (E = 3.5)
mean of 1000000 die rolls: 3.502   (E = 3.5)

9. Variance and covariance

Variance

The expected squared distance from the mean — the expectation of $g(x) = (x - \mathbb E [X])^2$:

$$ \operatorname{var}(X) = \mathbb{E}\bigl[(X - \mathbb E[X])^2\bigr] = \mathbb E[X^2] - \mathbb E[X]^2 $$

The shortcut on the right is derived on the next slide. $\;\sigma = \sqrt{\operatorname{var}(X)}$ is the standard deviation, in the unit of $X$.

Why the shortcut holds

With $\mu = \mathbb E[X]$ — a number, not random:

$$ \begin{aligned} \mathbb E\bigl[(X - \mu)^2\bigr] &= \mathbb E\bigl[X^2 - 2\mu X + \mu^2\bigr] && \text{expand} \\ &= \mathbb E[X^2] - 2\mu\,\mathbb E[X] + \mu^2 && \text{linearity} \\ &= \mathbb E[X^2] - \mu^2 && \mathbb E[X] = \mu \end{aligned} $$

Bernoulli: $X^2 = X$ (because $0^2 = 0$ and $1^2 = 1$), so $\mathbb E[X^2] = \theta$ and

$$ \operatorname{var}(X) = \theta - \theta^2 = \theta(1 - \theta) $$

— largest at $\theta = 0.5$, zero for a certain outcome.

Rule 1: constants

$$ \begin{aligned} \operatorname{var}(aX + b) &= \mathbb E\bigl[(aX + b - a\mu - b)^2\bigr] && \mathbb E[aX + b] = a\mu + b \\ &= a^2\,\mathbb E\bigl[(X - \mu)^2\bigr] && \text{linearity} \\ &= a^2 \operatorname{var}(X) \end{aligned} $$

A shift does nothing; a factor comes out squared.

Rule 2: sums, and the covariance

The covariance measures how $X$ and $Y$ deviate from their means together:

$$ \operatorname{Cov}(X, Y) = \mathbb E\bigl[(X - \mu_X)(Y - \mu_Y)\bigr] = \mathbb E[XY] - \mu_X\,\mu_Y, \qquad \operatorname{Cov}(X, X) = \operatorname{var}(X) $$

The shortcut follows like the variance's (derivation in the notes).

$$ \begin{aligned} \operatorname{var}(X + Y) &= \mathbb E\bigl[\bigl((X - \mu_X) + (Y - \mu_Y)\bigr)^2\bigr] && \text{since } \mathbb E[X + Y] = \mu_X + \mu_Y \\ &= \mathbb E\bigl[(X - \mu_X)^2\bigr] + \mathbb E\bigl[(Y - \mu_Y)^2\bigr] + 2\,\mathbb E\bigl[(X - \mu_X)(Y - \mu_Y)\bigr] && \text{expand, linearity} \\ &= \operatorname{var}(X) + \operatorname{var}(Y) + 2\operatorname{Cov}(X, Y) \end{aligned} $$

Likewise $\operatorname{var}(X - Y) = \operatorname{var}(X) + \operatorname{var}(Y) - 2\operatorname{Cov}(X, Y)$.

The covariance shortcut.

$$ \begin{aligned} \mathbb E\bigl[(X - \mu_X)(Y - \mu_Y)\bigr] &= \mathbb E\bigl[XY - \mu_Y X - \mu_X Y + \mu_X\mu_Y\bigr] && \text{expand} \\ &= \mathbb E[XY] - \mu_Y\mu_X - \mu_X\mu_Y + \mu_X\mu_Y && \text{linearity} \\ &= \mathbb E[XY] - \mu_X\mu_Y \end{aligned} $$

Independence removes the covariance

$$ \begin{aligned} \mathbb E[XY] &= \textstyle\sum_{x, y} x y\, p(x)\,p(y) && X \perp Y \\ &= \textstyle\bigl(\sum_x x\,p(x)\bigr)\bigl(\sum_y y\,p(y)\bigr) = \mathbb E[X]\,\mathbb E[Y] \end{aligned} $$

so $\operatorname{Cov}(X, Y) = 0$ and variances of independent variables add.

The converse is false: let $X$ be uniform on $\{-1, 0, 1\}$ and $Y = X^2$. Then $\mathbb E[X] = 0$ and $XY = X^3$ takes the values $-1, 0, 1$ with equal probability, so $\mathbb E[XY] = 0$ and $\operatorname{Cov}(X, Y) = 0 - 0 \cdot \mathbb E[Y] = 0$ — yet $Y$ is a function of $X$. Covariance measures only linear dependence.

i.i.d.

Data $X_1, \dots, X_n$ are i.i.d. — independent and identically distributed — if they are mutually independent and all have the same distribution, with variance $\sigma^2$.

$$ \begin{aligned} \operatorname{var}\Bigl(\sum_{i=1}^n X_i\Bigr) &= \sum_{i=1}^n \operatorname{var}(X_i) = n\sigma^2 && \text{Rule 2, all covariances } 0 \\ \operatorname{var}(\bar X) &= \frac{1}{n^2}\, n\sigma^2 = \frac{\sigma^2}{n} && \text{Rule 1 with } a = \tfrac1n \end{aligned} $$

So the typical distance of a mean from $\mu$ is $\sigma/\sqrt n$ — the $1/\sqrt n$ of §3. The recap Estimators derives it as the variance of the sample mean, and Uncertainty turns it into an error bar; its paired comparison of two models lives on the $-2\operatorname{Cov}$ term of Rule 2.

In [11]:
a = rng.normal(0, 1, 200_000)
b = 0.8 * a + 0.6 * rng.normal(0, 1, 200_000)       # correlated with a, var(b) = 1
for name, x, y in [("independent", a, rng.normal(0, 1, 200_000)), ("correlated", a, b)]:
    cov = np.mean(x * y) - x.mean() * y.mean()
    print(f"{name:12} var(x+y) = {np.var(x + y):.3f}   var x + var y + 2 cov = "
          f"{np.var(x) + np.var(y) + 2 * cov:.3f}   (cov = {cov:+.3f})")
independent  var(x+y) = 2.007   var x + var y + 2 cov = 2.007   (cov = +0.000)
correlated   var(x+y) = 3.619   var x + var y + 2 cov = 3.619   (cov = +0.805)

10. The binomial distribution

$n$ i.i.d. trials $X_i \sim \mathrm{Bernoulli}(\theta)$; the number of successes $K = X_1 + \dots + X_n$ takes the values $k = 0, \dots, n$ with

$$ p(K{=}k) = \binom{n}{k}\,\theta^{k}(1-\theta)^{n-k}, \qquad K \sim \mathrm{Binomial}(n, \theta) $$

$\theta^k(1-\theta)^{n-k}$ is the probability of one sequence with $k$ successes (independence); $\binom nk$ counts the sequences.

Mean and variance — no new work

$$ \begin{aligned} \mathbb E[K] &= \textstyle\sum_i \mathbb E[X_i] = n\theta && \text{linearity (§8)} \\ \operatorname{var}(K) &= \textstyle\sum_i \operatorname{var}(X_i) = n\theta(1-\theta) && \text{independent sum (§9)} \end{aligned} $$

Example: a classifier with true accuracy $\theta = 0.7$ on $n = 50$ test examples gets on average $35$ right, give or take $\sqrt{50 \cdot 0.21} \approx 3.2$.

In [12]:
n_tr, theta = 50, 0.7
K = (rng.random((100_000, n_tr)) < theta).sum(axis=1)   # 100 000 test sets
ks = np.arange(n_tr + 1)
pmf = np.array([math.comb(n_tr, k) * theta**k * (1 - theta)**(n_tr - k) for k in ks])

fig, ax = plt.subplots(figsize=(9, 3.2))
ax.hist(K, bins=np.arange(-0.5, n_tr + 1), density=True, color="steelblue", label="simulated")
ax.plot(ks, pmf, "ko-", ms=3, label="binomial pmf")
ax.set_xlim(20, 50); ax.set_xlabel("k correct out of 50"); ax.legend(); plt.show()
No description has been provided for this image
In [13]:
print(f"mean {K.mean():.2f} (n theta = {n_tr * theta:.2f})   "
      f"var {K.var():.2f} (n theta (1 - theta) = {n_tr * theta * (1 - theta):.2f})")
mean 35.00 (n theta = 35.00)   var 10.48 (n theta (1 - theta) = 10.50)

Divide by $n$: the accuracy $K/n$ has mean $\theta$ and variance $\theta(1-\theta)/n$ — the starting point of the error bars in the recap Uncertainty.

11. Continuous variables and the Gaussian

Densities

A height, a noise term $\epsilon$, a feature value: real numbers. Any single value has probability $0$ — the integral over $[a, a]$ vanishes. Probabilities come from intervals:

$$ p(a \le X \le b) = \int_a^b p(x)\,\mathrm dx, \qquad p(x) \ge 0, \qquad \int_{-\infty}^{\infty} p(x)\,\mathrm dx = 1 $$

$p(x)$ is a density — probability per unit of $x$. It can be larger than 1.

Sums become integrals, and every rule of §8–§9 carries over:

$$ \mathbb E[g(X)] = \int g(x)\,p(x)\,\mathrm dx, \qquad \mathbb E[X] = \int x\,p(x)\,\mathrm dx, \qquad \operatorname{var}(X) = \int (x - \mathbb E X)^2\,p(x)\,\mathrm dx $$

The Gaussian

$$ p(x) = \mathcal N(x \mid \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right), \qquad \mathbb E[X] = \mu, \quad \operatorname{var}(X) = \sigma^2 $$

Mean and variance follow by integration; we take them as given. The default noise model ($y = t(\vec x) + \epsilon$; → L-BiasVar), and — by the central limit theorem (Uncertainty) — the shape of nearly every average.

In [14]:
def gauss_pdf(x, mu=0.0, sigma=1.0):
    return np.exp(-(x - mu)**2 / (2 * sigma**2)) / np.sqrt(2 * np.pi * sigma**2)

xs = np.linspace(-4, 4, 400)
draws = rng.normal(0, 1, 100_000)
fig, ax = plt.subplots(figsize=(9, 3.2))
ax.hist(draws, bins=100, density=True, color="steelblue", label="100 000 draws")
ax.plot(xs, gauss_pdf(xs), "k", lw=2, label=r"$\mathcal{N}(0, 1)$")
for s in (-1.96, 1.96): ax.axvline(s, color="red", ls="--")
ax.legend(); ax.set_title("standard Gaussian; red: ±1.96"); plt.show()
No description has been provided for this image

Two numbers to remember

interval probability (integral of the density, computed below)
$\mu \pm \sigma$ 68.3 %
$\mu \pm 1.96\,\sigma$ 95.0 %

The $1.96$ is where the "95 % confidence interval" comes from (Uncertainty).

In [15]:
for z in [1.0, 1.96]:
    exact = math.erf(z / math.sqrt(2))       # p(|X| <= z) for X ~ N(0, 1)
    print(f"p(|X| <= {z:4}) = {exact:.4f}   simulated: {np.mean(np.abs(draws) <= z):.4f}")
p(|X| <=  1.0) = 0.6827   simulated: 0.6862
p(|X| <= 1.96) = 0.9500   simulated: 0.9508

math.erf is the error function $\operatorname{erf}(u) = \frac{2}{\sqrt\pi}\int_0^u e^{-t^2}\mathrm dt$; for the standard Gaussian $p(|X| \le z) = \operatorname{erf}(z/\sqrt2)$. A density of $\mathcal N(0, 0.1^2)$ peaks at $1/\sqrt{2\pi \cdot 0.01} \approx 4$ — a density above 1, while every probability stays below 1.

12. Summary

Take away (1/2)

  1. Three axioms — everything else (complement, marginals, Bayes) is derived.
  2. A random variable attaches numbers to outcomes; $\mathrm{Bernoulli}(\theta)$ models one right/wrong prediction.
  3. Conditional probability restricts the sample space; $p(A \mid B)$ and $p(B \mid A)$ are different numbers.
  4. Bayes' rule reverses the condition — and needs the base rate.
  5. Independence and conditional independence are different statements.

Take away (2/2)

  1. $\mathbb E[g(X)] = \sum_x g(x)\,p(x)$ — but $\mathbb E[g(X)] \ne g(\mathbb E X)$ in general. $\mathbb E$ is linear, always. $\operatorname{var}(aX + b) = a^2\operatorname{var}(X)$.
  2. $\operatorname{var}(X \pm Y) = \operatorname{var}X + \operatorname{var}Y \pm 2\operatorname{Cov}(X, Y)$; for independent variables the covariance vanishes.
  3. Binomial: $n\theta$ and $n\theta(1-\theta)$ — the accuracy $K/n$ scatters with variance $\theta(1-\theta)/n$.
  4. A density is not a probability; $\mu \pm 1.96\,\sigma$ holds 95 % of a Gaussian.
  5. The law of large numbers is why counting in a simulation works.

Exercises and Lab

fundamentals-Probability-exercises.ipynb — pen and paper: two dice, a table of test results read as a joint distribution, Bayes' rule and the base rate, the rules for expectation and variance, covariance and independence.

fundamentals-Probability-lab.ipynb — programming: each rule checked by simulation — the law of large numbers, conditional probabilities by counting, Bayes' rule, the binomial, the Gaussian density, the variance of a dependent sum.

Literature