Machine Learning · Fundamentals · Probability
Pen-and-paper exercises on events, joint and conditional probabilities, Bayes' rule and the rules
for expectation, variance and covariance. The programming exercises are in
fundamentals-Probability-lab.ipynb. References point back to the recap Probability.
Solutions are in skip cells (tag solution), stripped for the handout.
Two fair dice are rolled. (Probability)
Task 1. Write down the sample space $\Omega$. How many outcomes does it have, and what is the probability of each?
Task 2. Compute $p(\text{sum} = 7)$ and $p(\text{sum} \ge 11)$.
Task 3. Compute $p(\text{at least one six})$ — once by counting, once via the complement.
A medical test is applied to $200$ patients. $Y = 1$: the patient is ill; $T = 1$: the test is positive. Divide by $200$ to get the joint distribution of $Y$ and $T$:
| $T = 1$ (positive) | $T = 0$ (negative) | |
|---|---|---|
| $Y = 1$ (ill) | 45 | 15 |
| $Y = 0$ (healthy) | 5 | 135 |
Task 1. Write down $p(Y, T)$ and both marginals $p(Y)$, $p(T)$ (Probability).
Task 2. From the joint table, compute (Probability)
Task 3. Are $Y$ and $T$ independent? What would independence mean for this test?
A test detects $95\,\%$ of the ill, $p(\text{pos} \mid \text{ill}) = 0.95$, and gives a false alarm for $2\,\%$ of the healthy, $p(\text{pos} \mid \text{healthy}) = 0.02$ (Probability).
Task 1. Compute $p(\text{ill} \mid \text{pos})$ for a prevalence of $p(\text{ill}) = 0.2$.
Task 2. The same test is used for population screening, prevalence $0.002$. Compute $p(\text{ill} \mid \text{pos})$ again.
Task 3. The test did not change. Explain in two sentences why $p(\text{ill} \mid \text{pos})$ collapsed, and what this means for a test that was only ever tried on a group where half the patients were ill.
Task 1. Prove $\operatorname{var}(aX + b) = a^2 \operatorname{var}(X)$ from the definition of the variance.
Task 2. Compute $\mathbb E[F]$, $\mathbb E[F^2]$ (with $\mathbb E[g(F)] = \sum_f g(f)\,p(f)$) and $\operatorname{var}(F)$ for the face $F$ of a fair die. Compare $\mathbb E[1/F]$ with $1/\mathbb E[F]$.
Task 3. $X_1, \dots, X_n$ are i.i.d. with variance $\sigma^2$. Show that $\operatorname{var}(\bar X) = \sigma^2/n$ for $\bar X = \frac1n \sum_i X_i$. Which property of the data did you use where?
Task 1. Show: if $X \perp Y$ (discrete), then $\operatorname{Cov}(X, Y) = 0$.
Task 2. Let $X$ be uniform on $\{-1, 0, 1\}$ and $Y = X^2$. Compute $\operatorname{Cov}(X, Y)$, and show that $X$ and $Y$ are not independent.
Task 3. Two classifiers are tested on the same examples; $c_A, c_B \in \{0, 1\}$ indicate correctness. Why do you expect $\operatorname{Cov}(c_A, c_B) > 0$, and what does that do to $\operatorname{var}(c_A - c_B)$?