Exercises

Machine Learning · Fundamentals · Probability

Pen-and-paper exercises on events, joint and conditional probabilities, Bayes' rule and the rules for expectation, variance and covariance. The programming exercises are in fundamentals-Probability-lab.ipynb. References point back to the recap Probability. Solutions are in skip cells (tag solution), stripped for the handout.

Exercise 1 — Two dice

Two fair dice are rolled. (Probability)

Task 1. Write down the sample space $\Omega$. How many outcomes does it have, and what is the probability of each?

Task 2. Compute $p(\text{sum} = 7)$ and $p(\text{sum} \ge 11)$.

Task 3. Compute $p(\text{at least one six})$ — once by counting, once via the complement.

Your answer — Exercise 1

Exercise 2 — A table of test results is a joint distribution

A medical test is applied to $200$ patients. $Y = 1$: the patient is ill; $T = 1$: the test is positive. Divide by $200$ to get the joint distribution of $Y$ and $T$:

$T = 1$ (positive) $T = 0$ (negative)
$Y = 1$ (ill) 45 15
$Y = 0$ (healthy) 5 135

Task 1. Write down $p(Y, T)$ and both marginals $p(Y)$, $p(T)$ (Probability).

Task 2. From the joint table, compute (Probability)

  • $p(T{=}1 \mid Y{=}1)$ — the share of the ill whom the test detects,
  • $p(Y{=}1 \mid T{=}1)$ — the share of the positive tests that belong to ill patients,
  • $p(T{=}1 \mid Y{=}0)$ — the share of the healthy with a false alarm.

Task 3. Are $Y$ and $T$ independent? What would independence mean for this test?

Your answer — Exercise 2

Exercise 3 — Bayes' rule and the base rate

A test detects $95\,\%$ of the ill, $p(\text{pos} \mid \text{ill}) = 0.95$, and gives a false alarm for $2\,\%$ of the healthy, $p(\text{pos} \mid \text{healthy}) = 0.02$ (Probability).

Task 1. Compute $p(\text{ill} \mid \text{pos})$ for a prevalence of $p(\text{ill}) = 0.2$.

Task 2. The same test is used for population screening, prevalence $0.002$. Compute $p(\text{ill} \mid \text{pos})$ again.

Task 3. The test did not change. Explain in two sentences why $p(\text{ill} \mid \text{pos})$ collapsed, and what this means for a test that was only ever tried on a group where half the patients were ill.

Your answer — Exercise 3

Exercise 4 — Rules for expectation and variance

(Probability)

Task 1. Prove $\operatorname{var}(aX + b) = a^2 \operatorname{var}(X)$ from the definition of the variance.

Task 2. Compute $\mathbb E[F]$, $\mathbb E[F^2]$ (with $\mathbb E[g(F)] = \sum_f g(f)\,p(f)$) and $\operatorname{var}(F)$ for the face $F$ of a fair die. Compare $\mathbb E[1/F]$ with $1/\mathbb E[F]$.

Task 3. $X_1, \dots, X_n$ are i.i.d. with variance $\sigma^2$. Show that $\operatorname{var}(\bar X) = \sigma^2/n$ for $\bar X = \frac1n \sum_i X_i$. Which property of the data did you use where?

Your answer — Exercise 4

Exercise 5 — Covariance and independence

Task 1. Show: if $X \perp Y$ (discrete), then $\operatorname{Cov}(X, Y) = 0$.

Task 2. Let $X$ be uniform on $\{-1, 0, 1\}$ and $Y = X^2$. Compute $\operatorname{Cov}(X, Y)$, and show that $X$ and $Y$ are not independent.

Task 3. Two classifiers are tested on the same examples; $c_A, c_B \in \{0, 1\}$ indicate correctness. Why do you expect $\operatorname{Cov}(c_A, c_B) > 0$, and what does that do to $\operatorname{var}(c_A - c_B)$?

Your answer — Exercise 5

Summary

You should be able to answer

  • Why is $p(\text{at least one six})$ easiest via the complement?
  • Why are $p(\text{pos} \mid \text{ill})$ and $p(\text{ill} \mid \text{pos})$ different numbers?
  • Why does $p(\text{ill} \mid \text{pos})$ depend on the base rate, and $p(\text{pos} \mid \text{ill})$ not?
  • Why does $\operatorname{var}(\bar X)$ shrink like $1/n$ — and which assumption does that need?
  • Does $\operatorname{Cov}(X, Y) = 0$ imply independence?
  • How can a density exceed 1?

Literature

  • L. Wasserman: All of Statistics, Springer 2004 — ch. 1–3
  • C. M. Bishop: Pattern Recognition and Machine Learning, Springer 2006 — §1.2