Machine Learning · Fundamentals
Events · random variables · conditional probability · Bayes' rule · expectation and variance · the binomial and the Gaussian
| Part | The question it answers |
|---|---|
| 1 · Outcome, event, sample space | Which rules does every probability obey? |
| 2 · Random variables | How is a number attached to a random result? |
| 3 · The law of large numbers | Why does counting in a simulation estimate a probability? |
| 4 · Joint and marginal | How do two random variables behave together, and each alone? |
| 5 · Conditional probability and independence | How does knowing one value change the other? |
| 6 · Bayes' rule | How do we turn $p(A \mid B)$ into $p(B \mid A)$? |
| 7 · Conditional independence | When does a third variable make two variables independent? |
| 8 · Expectation · 9 · Variance and covariance | What is the average value, and how much does it scatter? |
| 10 · The binomial distribution | How many successes in $n$ trials? |
| 11 · Continuous variables and the Gaussian | What replaces probabilities when the values are real numbers? |
import math
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
np.set_printoptions(precision=4, suppress=True)
plt.rcParams["figure.figsize"] = (9, 4)
plt.rcParams["axes.grid"] = True
plt.rcParams["grid.alpha"] = 0.3
rng = np.random.default_rng(0)
Method: derivation first, simulation second. Every rule is derived from the axioms; a simulation then checks it by counting.
Notation: $p(\cdot)$ — lower case — for every probability, probability mass and density. Many books write $P$.
A probability $p$ assigns a number to every event $A$ such that
Wasserman 2004, Def. 1.5. Everything else in this recap is derived from these three.
An event's probability is the sum over its outcomes — the single outcomes are disjoint, so axiom 3 applies: $\;p(A) = \sum_{\omega \in A} p(\{\omega\})$.
Fair die: six equal $p(\{\omega\}) = q$ with $6q = p(\Omega) = 1$, so $q = 1/6$ and $p(\text{even}) = 3/6$.
The complement $A^c = \Omega \setminus A$ is disjoint from $A$, and $A \cup A^c = \Omega$:
$$ p(A^c) = 1 - p(A) $$
Often the easy way round: $p(\text{at least one six in two rolls}) = 1 - p(\text{no six}) = 1 - (5/6)^2$. Each roll shows no six with probability $5/6$; the two probabilities multiply because the rolls are independent (§5).
rolls = rng.integers(1, 7, size=10_000) # 10 000 rolls of a fair die
faces = np.arange(1, 7)
freq = np.array([np.mean(rolls == f) for f in faces])
print("relative frequency of each face:", freq.round(3), " (1/6 = 0.167)")
print(f"relative frequency of 'even' : {np.mean(rolls % 2 == 0):.3f} (3/6 = 0.5)")
A random variable is a function $X: \Omega \to \mathbb R$ that attaches a number to every outcome. Capital letters for the variable, lower case for its values.
Its set of possible values is $\operatorname{Val}(X)$.
Die: $X = 1$ if the face is odd, else $0$; $\;\operatorname{Val}(X) = \{0, 1\}$.
$\{X = x\}$ is short for the event $\{\omega \in \Omega : X(\omega) = x\}$, so
$$ p(X{=}1) = p(\{1, 3, 5\}) = \tfrac12 . $$
Notation: $p(X)$ is the whole distribution — a table or a function; $p(X{=}x)$ is one number; $p(x)$ is short for $p(X{=}x)$ when unambiguous.
A discrete distribution assigns a probability to each value:
$$ p(X{=}x) \ge 0 \qquad\text{and}\qquad \sum_{x \in \operatorname{Val}(X)} p(X{=}x) = 1 $$
Both follow from the axioms: the events $\{X = x\}$ for different $x$ are disjoint and together form $\Omega$.
Two values, $1$ ("success") and $0$ ("failure"):
$$ p(X{=}1) = \theta, \qquad p(X{=}0) = 1 - \theta, \qquad X \sim \mathrm{Bernoulli}(\theta) $$
Repeat an experiment $n$ times independently and count how often event $A$ occurs. As $n$ grows, the relative frequency approaches $p(A)$.
More generally: the mean of $n$ independent repetitions of a random variable approaches its expected value (§8). Wasserman 2004, Thm. 5.6.
This is why every simulation in this course can check a probability by counting.
even = (rolls % 2 == 0)
running = np.cumsum(even) / np.arange(1, len(rolls) + 1)
fig, ax = plt.subplots(figsize=(9, 3.2))
ax.plot(np.arange(1, len(rolls) + 1), running, lw=1.5)
ax.axhline(0.5, color="k", ls=":", label=r"$p(\mathrm{even}) = 0.5$")
ax.set_xscale("log"); ax.set_xlabel("number of rolls n"); ax.set_ylim(0, 1)
ax.set_title("running relative frequency of 'even'"); ax.legend(); plt.show()
The frequency settles — but slowly: its typical distance from $0.5$ shrinks like $1/\sqrt n$ — §9 shows why, and the recap Uncertainty turns it into an error bar.
$X = 1$ if the face is odd, $Y = \mathbb 1[\text{face} > 3]$ — the indicator: $1$ if the condition holds, else $0$.
The joint distribution $p(X, Y)$ gives every combination:
| $p(X, Y)$ | $Y = 0$ (faces 1–3) | $Y = 1$ (faces 4–6) |
|---|---|---|
| $X = 1$ (odd) | $2/6$ — faces 1, 3 | $1/6$ — face 5 |
| $X = 0$ (even) | $1/6$ — face 2 | $2/6$ — faces 4, 6 |
The distribution of one variable: sum the joint over the other.
$$ p(X{=}x) = \sum_{y} p(X{=}x, Y{=}y) $$
The events $\{X{=}x, Y{=}y\}$ for different $y$ are disjoint and together give $\{X = x\}$.
Die: $p(X{=}1) = 2/6 + 1/6 = 1/2$ — the row sum. The column sums give $p(Y)$.
X = (rolls % 2 == 1).astype(int) # odd
Y = (rolls > 3).astype(int) # greater than 3
joint = np.array([[np.mean((X == x) & (Y == y)) for y in (0, 1)] for x in (0, 1)])
print("estimated p(X=x, Y=y) (rows x = 0, 1; columns y = 0, 1)")
print(joint.round(3), " exact: [[1/6, 2/6], [2/6, 1/6]]")
print("marginal p(X):", joint.sum(axis=1).round(3), " marginal p(Y):", joint.sum(axis=0).round(3))
$$ p(X \mid Y) = \frac{p(X, Y)}{p(Y)}, \qquad p(Y) > 0 $$
"Probability of $X$ given $Y$". Conditioning restricts the sample space to the outcomes with $Y = y$ and rescales them to sum to 1.
Die: $p(X{=}1 \mid Y{=}1) = \dfrac{1/6}{1/2} = \dfrac13$ — of the faces 4, 5, 6 only 5 is odd.
kept = rolls[Y == 1] # keep only the rolls with face > 3
print(f"faces kept: {np.unique(kept)}")
print(f"share of odd faces among them: {np.mean(kept % 2 == 1):.3f} (1/3 = 0.333)")
Multiply the definition by $p(Y)$:
$$ p(X, Y) = p(X \mid Y)\,p(Y) = p(Y \mid X)\,p(X) $$
Both ways round — the second line of §6 starts here.
$X$ and $Y$ are independent, $X \perp Y$, if for all values
$$ p(X, Y) = p(X)\,p(Y) $$
Equivalently, knowing $Y$ does not change the distribution of $X$:
$$ p(X \mid Y) = \frac{p(X, Y)}{p(Y)} = \frac{p(X)\,p(Y)}{p(Y)} = p(X) $$
One die: $p(X{=}1, Y{=}1) = 1/6 \ne p(X{=}1)\,p(Y{=}1) = 1/4$ — dependent: knowing "greater than 3" makes "odd" less likely.
Two fair dice, $X$ from the first and $Y$ from the second: all $36$ pairs of faces are equally likely, so $p(i, j) = \frac{1}{36} = \frac16 \cdot \frac16 = p(i)\,p(j)$ — independent, and so are $X$ and $Y$.
second = rng.integers(1, 7, size=len(rolls)) # a second, separate die
Y2 = (second > 3).astype(int)
print(f"one die : p(X=1, Y=1) = {np.mean((X == 1) & (Y == 1)):.3f} "
f"p(X=1) p(Y=1) = {X.mean() * Y.mean():.3f}")
print(f"two dice: p(X=1, Y=1) = {np.mean((X == 1) & (Y2 == 1)):.3f} "
f"p(X=1) p(Y=1) = {X.mean() * Y2.mean():.3f}")
$$ \begin{aligned} p(B \mid A) &= \frac{p(A, B)}{p(A)} && \text{definition} \\ &= \frac{p(A \mid B)\,p(B)}{p(A)} && \text{product rule} \end{aligned} $$
and the denominator by marginalisation (§4) and the product rule:
$$ p(A) = \sum_{b} p(A, B{=}b) = \sum_{b} p(A \mid B{=}b)\,p(B{=}b) $$
A patient tests positive. How likely is the disease, $p(\text{ill} \mid \text{pos})$?
$$ p(\text{ill} \mid \text{pos}) = \frac{0.9 \cdot 0.01}{0.9 \cdot 0.01 + 0.05 \cdot 0.99} = \frac{0.009}{0.0585} \approx 0.15 $$
A "90 % test" — and 85 % of the positive tests belong to healthy people. The healthy are so many that their 5 % false alarms outnumber the true cases: the base-rate fallacy.
n_people = 1_000_000
ill = rng.random(n_people) < 0.01
pos = np.where(ill, rng.random(n_people) < 0.9, rng.random(n_people) < 0.05)
print(f"share of the positive tests that are ill: {ill[pos].mean():.3f}")
print(f"Bayes' rule : {0.009 / (0.009 + 0.05 * 0.99):.3f}")
$p(\text{pos} \mid \text{ill})$ and $p(\text{pos} \mid \text{healthy})$ are properties of the test; $p(\text{ill} \mid \text{pos})$ depends on the base rate as well.
A classifier is a test of exactly this kind. The evaluation metrics give these conditional probabilities their names (→ L-Eval), and the fairness metrics compare base rates between groups (→ L-Fair).
$X$ and $Y$ are conditionally independent given $Z$, $X \perp Y \mid Z$, if
$$ p(X, Y \mid Z) = p(X \mid Z)\,p(Y \mid Z) \qquad\Longleftrightarrow\qquad p(X \mid Y, Z) = p(X \mid Z) $$
Once $Z$ is known, $Y$ carries no further information about $X$.
Why the two forms are equivalent.
$$ \begin{aligned} p(X \mid Y, Z) &= \frac{p(X, Y \mid Z)}{p(Y \mid Z)} && \text{definition, everything given } Z \\ &= \frac{p(X \mid Z)\,p(Y \mid Z)}{p(Y \mid Z)} && \text{first form} \\ &= p(X \mid Z) \end{aligned} $$
Read backwards, with the product rule, the second form gives the first.
Pick a coin at random: fair ($Z = 0$, heads $0.5$) or biased ($Z = 1$, heads $0.9$). Toss it twice; $X$, $Y$ = heads on toss 1, 2. Given the coin, the tosses are independent.
$$ \begin{aligned} p(X{=}1) &= \textstyle\sum_z p(z)\,p(X{=}1 \mid z) = \tfrac12 \cdot 0.5 + \tfrac12 \cdot 0.9 = 0.7 && \text{marginalise over } Z \\ p(X{=}1, Y{=}1) &= \textstyle\sum_z p(z)\,p(X{=}1 \mid z)\,p(Y{=}1 \mid z) = \tfrac12 \cdot 0.5^2 + \tfrac12 \cdot 0.9^2 = 0.53 && \text{and } X \perp Y \mid Z \end{aligned} $$
$0.53 \ne 0.7 \cdot 0.7$: without the coin, the tosses are dependent.
A first head hints at the biased coin, and so at a second head.
reps = 100_000
Z = rng.integers(0, 2, size=reps) # which coin
heads = np.where(Z == 1, 0.9, 0.5)
Xc = (rng.random(reps) < heads).astype(int) # toss 1
Yc = (rng.random(reps) < heads).astype(int) # toss 2, same coin
print(f"without Z : p(X=1,Y=1) = {np.mean(Xc & Yc):.3f} p(X=1)p(Y=1) = {Xc.mean() * Yc.mean():.3f}")
for z in (0, 1):
m = Z == z
print(f"given Z={z} : p(X=1,Y=1|Z) = {np.mean(Xc[m] & Yc[m]):.3f} "
f"p(X=1|Z)p(Y=1|Z) = {Xc[m].mean() * Yc[m].mean():.3f}")
Independence and conditional independence do not imply each other.
The expected value is the probability-weighted average of the values:
$$ \mathbb{E}[X] = \sum_{x \in \operatorname{Val}(X)} x\, p(X{=}x) $$
Bernoulli: $\mathbb E[X] = 1\cdot\theta + 0\cdot(1-\theta) = \theta$. Fair die: $\mathbb E[\text{face}] = \frac16(1 + \dots + 6) = 3.5$ — not a possible face.
Often we need the average of a function of $X$ — a square, a loss, a logarithm. $g(X)$ is itself a random variable, and its expectation needs no new distribution:
$$ \mathbb{E}[g(X)] = \sum_{x \in \operatorname{Val}(X)} g(x)\, p(X{=}x) $$
Weight $g$ of each value with the probability of that value. For two variables, likewise $\;\mathbb{E}[g(X, Y)] = \sum_{x, y} g(x, y)\, p(x, y)$.
Die: $\mathbb E[F^2] = \frac16(1 + 4 + 9 + 16 + 25 + 36) = \frac{91}{6} \approx 15.17$.
In general $\mathbb E[g(X)] \ne g(\mathbb E[X])$: here $(\mathbb E F)^2 = 3.5^2 = 12.25$. The gap, $15.17 - 12.25$, is exactly the variance of §9.
Everything that follows is of this form: the variance $\mathbb E[(X - \mu)^2]$, the covariance $\mathbb E[(X - \mu_X)(Y - \mu_Y)]$, and the mean squared error $\mathbb E[(\hat\theta - \theta)^2]$.
F = rng.integers(1, 7, size=100_000)
print(f"mean of F^2 over 100 000 rolls: {np.mean(F**2):.3f} (E[F^2] = 91/6 = {91 / 6:.3f})")
print(f"(mean of F)^2 : {F.mean()**2:.3f} ((E F)^2 = 12.25)")
For random variables $X, Y$ and constants $a, b$ — dependent or not:
$$ \begin{aligned} \mathbb{E}[aX + bY] &= \textstyle\sum_{x, y} (ax + by)\, p(x, y) && \mathbb E[g(X, Y)] \\ &= a \textstyle\sum_x x \sum_y p(x, y) + b \sum_y y \sum_x p(x, y) && \text{split the sum} \\ &= a\,\mathbb{E}[X] + b\,\mathbb{E}[Y] && \text{inner sums are marginals (§4)} \end{aligned} $$
Consequence: the sample mean $\bar X = \frac1n\sum_i X_i$ of variables with $\mathbb E[X_i] = \mu$ has $\mathbb E[\bar X] = \mu$. The law of large numbers (§3) says more: for independent $X_i$, $\bar X$ itself approaches $\mu$.
for n in [10, 1_000, 100_000, 1_000_000]:
print(f"mean of {n:6} die rolls: {rng.integers(1, 7, size=n).mean():.3f} (E = 3.5)")
The expected squared distance from the mean — the expectation of $g(x) = (x - \mathbb E [X])^2$:
$$ \operatorname{var}(X) = \mathbb{E}\bigl[(X - \mathbb E[X])^2\bigr] = \mathbb E[X^2] - \mathbb E[X]^2 $$
The shortcut on the right is derived on the next slide. $\;\sigma = \sqrt{\operatorname{var}(X)}$ is the standard deviation, in the unit of $X$.
With $\mu = \mathbb E[X]$ — a number, not random:
$$ \begin{aligned} \mathbb E\bigl[(X - \mu)^2\bigr] &= \mathbb E\bigl[X^2 - 2\mu X + \mu^2\bigr] && \text{expand} \\ &= \mathbb E[X^2] - 2\mu\,\mathbb E[X] + \mu^2 && \text{linearity} \\ &= \mathbb E[X^2] - \mu^2 && \mathbb E[X] = \mu \end{aligned} $$
Bernoulli: $X^2 = X$ (because $0^2 = 0$ and $1^2 = 1$), so $\mathbb E[X^2] = \theta$ and
$$ \operatorname{var}(X) = \theta - \theta^2 = \theta(1 - \theta) $$
— largest at $\theta = 0.5$, zero for a certain outcome.
$$ \begin{aligned} \operatorname{var}(aX + b) &= \mathbb E\bigl[(aX + b - a\mu - b)^2\bigr] && \mathbb E[aX + b] = a\mu + b \\ &= a^2\,\mathbb E\bigl[(X - \mu)^2\bigr] && \text{linearity} \\ &= a^2 \operatorname{var}(X) \end{aligned} $$
A shift does nothing; a factor comes out squared.
The covariance measures how $X$ and $Y$ deviate from their means together:
$$ \operatorname{Cov}(X, Y) = \mathbb E\bigl[(X - \mu_X)(Y - \mu_Y)\bigr] = \mathbb E[XY] - \mu_X\,\mu_Y, \qquad \operatorname{Cov}(X, X) = \operatorname{var}(X) $$
The shortcut follows like the variance's (derivation in the notes).
$$ \begin{aligned} \operatorname{var}(X + Y) &= \mathbb E\bigl[\bigl((X - \mu_X) + (Y - \mu_Y)\bigr)^2\bigr] && \text{since } \mathbb E[X + Y] = \mu_X + \mu_Y \\ &= \mathbb E\bigl[(X - \mu_X)^2\bigr] + \mathbb E\bigl[(Y - \mu_Y)^2\bigr] + 2\,\mathbb E\bigl[(X - \mu_X)(Y - \mu_Y)\bigr] && \text{expand, linearity} \\ &= \operatorname{var}(X) + \operatorname{var}(Y) + 2\operatorname{Cov}(X, Y) \end{aligned} $$
Likewise $\operatorname{var}(X - Y) = \operatorname{var}(X) + \operatorname{var}(Y) - 2\operatorname{Cov}(X, Y)$.
The covariance shortcut.
$$ \begin{aligned} \mathbb E\bigl[(X - \mu_X)(Y - \mu_Y)\bigr] &= \mathbb E\bigl[XY - \mu_Y X - \mu_X Y + \mu_X\mu_Y\bigr] && \text{expand} \\ &= \mathbb E[XY] - \mu_Y\mu_X - \mu_X\mu_Y + \mu_X\mu_Y && \text{linearity} \\ &= \mathbb E[XY] - \mu_X\mu_Y \end{aligned} $$
$$ \begin{aligned} \mathbb E[XY] &= \textstyle\sum_{x, y} x y\, p(x)\,p(y) && X \perp Y \\ &= \textstyle\bigl(\sum_x x\,p(x)\bigr)\bigl(\sum_y y\,p(y)\bigr) = \mathbb E[X]\,\mathbb E[Y] \end{aligned} $$
so $\operatorname{Cov}(X, Y) = 0$ and variances of independent variables add.
The converse is false: let $X$ be uniform on $\{-1, 0, 1\}$ and $Y = X^2$. Then $\mathbb E[X] = 0$ and $XY = X^3$ takes the values $-1, 0, 1$ with equal probability, so $\mathbb E[XY] = 0$ and $\operatorname{Cov}(X, Y) = 0 - 0 \cdot \mathbb E[Y] = 0$ — yet $Y$ is a function of $X$. Covariance measures only linear dependence.
Data $X_1, \dots, X_n$ are i.i.d. — independent and identically distributed — if they are mutually independent and all have the same distribution, with variance $\sigma^2$.
$$ \begin{aligned} \operatorname{var}\Bigl(\sum_{i=1}^n X_i\Bigr) &= \sum_{i=1}^n \operatorname{var}(X_i) = n\sigma^2 && \text{Rule 2, all covariances } 0 \\ \operatorname{var}(\bar X) &= \frac{1}{n^2}\, n\sigma^2 = \frac{\sigma^2}{n} && \text{Rule 1 with } a = \tfrac1n \end{aligned} $$
So the typical distance of a mean from $\mu$ is $\sigma/\sqrt n$ — the $1/\sqrt n$ of §3. The recap Estimators derives it as the variance of the sample mean, and Uncertainty turns it into an error bar; its paired comparison of two models lives on the $-2\operatorname{Cov}$ term of Rule 2.
a = rng.normal(0, 1, 200_000)
b = 0.8 * a + 0.6 * rng.normal(0, 1, 200_000) # correlated with a, var(b) = 1
for name, x, y in [("independent", a, rng.normal(0, 1, 200_000)), ("correlated", a, b)]:
cov = np.mean(x * y) - x.mean() * y.mean()
print(f"{name:12} var(x+y) = {np.var(x + y):.3f} var x + var y + 2 cov = "
f"{np.var(x) + np.var(y) + 2 * cov:.3f} (cov = {cov:+.3f})")
$n$ i.i.d. trials $X_i \sim \mathrm{Bernoulli}(\theta)$; the number of successes $K = X_1 + \dots + X_n$ takes the values $k = 0, \dots, n$ with
$$ p(K{=}k) = \binom{n}{k}\,\theta^{k}(1-\theta)^{n-k}, \qquad K \sim \mathrm{Binomial}(n, \theta) $$
$\theta^k(1-\theta)^{n-k}$ is the probability of one sequence with $k$ successes (independence); $\binom nk$ counts the sequences.
$$ \begin{aligned} \mathbb E[K] &= \textstyle\sum_i \mathbb E[X_i] = n\theta && \text{linearity (§8)} \\ \operatorname{var}(K) &= \textstyle\sum_i \operatorname{var}(X_i) = n\theta(1-\theta) && \text{independent sum (§9)} \end{aligned} $$
Example: a classifier with true accuracy $\theta = 0.7$ on $n = 50$ test examples gets on average $35$ right, give or take $\sqrt{50 \cdot 0.21} \approx 3.2$.
n_tr, theta = 50, 0.7
K = (rng.random((100_000, n_tr)) < theta).sum(axis=1) # 100 000 test sets
ks = np.arange(n_tr + 1)
pmf = np.array([math.comb(n_tr, k) * theta**k * (1 - theta)**(n_tr - k) for k in ks])
fig, ax = plt.subplots(figsize=(9, 3.2))
ax.hist(K, bins=np.arange(-0.5, n_tr + 1), density=True, color="steelblue", label="simulated")
ax.plot(ks, pmf, "ko-", ms=3, label="binomial pmf")
ax.set_xlim(20, 50); ax.set_xlabel("k correct out of 50"); ax.legend(); plt.show()
print(f"mean {K.mean():.2f} (n theta = {n_tr * theta:.2f}) "
f"var {K.var():.2f} (n theta (1 - theta) = {n_tr * theta * (1 - theta):.2f})")
Divide by $n$: the accuracy $K/n$ has mean $\theta$ and variance $\theta(1-\theta)/n$ — the starting point of the error bars in the recap Uncertainty.
A height, a noise term $\epsilon$, a feature value: real numbers. Any single value has probability $0$ — the integral over $[a, a]$ vanishes. Probabilities come from intervals:
$$ p(a \le X \le b) = \int_a^b p(x)\,\mathrm dx, \qquad p(x) \ge 0, \qquad \int_{-\infty}^{\infty} p(x)\,\mathrm dx = 1 $$
$p(x)$ is a density — probability per unit of $x$. It can be larger than 1.
Sums become integrals, and every rule of §8–§9 carries over:
$$ \mathbb E[g(X)] = \int g(x)\,p(x)\,\mathrm dx, \qquad \mathbb E[X] = \int x\,p(x)\,\mathrm dx, \qquad \operatorname{var}(X) = \int (x - \mathbb E X)^2\,p(x)\,\mathrm dx $$
$$ p(x) = \mathcal N(x \mid \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right), \qquad \mathbb E[X] = \mu, \quad \operatorname{var}(X) = \sigma^2 $$
Mean and variance follow by integration; we take them as given. The default noise model ($y = t(\vec x) + \epsilon$; → L-BiasVar), and — by the central limit theorem (Uncertainty) — the shape of nearly every average.
def gauss_pdf(x, mu=0.0, sigma=1.0):
return np.exp(-(x - mu)**2 / (2 * sigma**2)) / np.sqrt(2 * np.pi * sigma**2)
xs = np.linspace(-4, 4, 400)
draws = rng.normal(0, 1, 100_000)
fig, ax = plt.subplots(figsize=(9, 3.2))
ax.hist(draws, bins=100, density=True, color="steelblue", label="100 000 draws")
ax.plot(xs, gauss_pdf(xs), "k", lw=2, label=r"$\mathcal{N}(0, 1)$")
for s in (-1.96, 1.96): ax.axvline(s, color="red", ls="--")
ax.legend(); ax.set_title("standard Gaussian; red: ±1.96"); plt.show()
| interval | probability (integral of the density, computed below) |
|---|---|
| $\mu \pm \sigma$ | 68.3 % |
| $\mu \pm 1.96\,\sigma$ | 95.0 % |
The $1.96$ is where the "95 % confidence interval" comes from (Uncertainty).
for z in [1.0, 1.96]:
exact = math.erf(z / math.sqrt(2)) # p(|X| <= z) for X ~ N(0, 1)
print(f"p(|X| <= {z:4}) = {exact:.4f} simulated: {np.mean(np.abs(draws) <= z):.4f}")
math.erf is the error function $\operatorname{erf}(u) = \frac{2}{\sqrt\pi}\int_0^u e^{-t^2}\mathrm dt$; for the standard
Gaussian $p(|X| \le z) = \operatorname{erf}(z/\sqrt2)$. A density of $\mathcal N(0, 0.1^2)$ peaks at
$1/\sqrt{2\pi \cdot 0.01} \approx 4$ — a density above 1, while every probability stays below 1.
fundamentals-Probability-exercises.ipynb — pen and paper: two dice, a table of test results read as a
joint distribution, Bayes' rule and the base rate, the rules for expectation and variance, covariance and
independence.
fundamentals-Probability-lab.ipynb — programming: each rule checked by simulation — the law of large
numbers, conditional probabilities by counting, Bayes' rule, the binomial, the Gaussian density, the
variance of a dependent sum.