{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "01771152",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Lab\n",
    "\n",
    "**Machine Learning** · Fundamentals · *Probability*\n",
    "\n",
    "The rules of the pen-and-paper exercises (`fundamentals-Probability-exercises.ipynb`), checked by\n",
    "simulation: a probability is a long-run relative frequency."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9843b84f",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## How to work through this\n",
    "\n",
    "The code is given — read it line by line. Only the lines marked `# TODO` are yours: they are the\n",
    "lines that carry the idea. Until you replace it, each `todo(...)` stops with an error that says what\n",
    "is missing. The self-checks test your `TODO` lines.\n",
    "\n",
    "After each block, answer the **questions** in the cell below it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1e90e7a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "-"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import math\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline\n",
    "np.set_printoptions(precision=4, suppress=True)\n",
    "plt.rcParams[\"figure.figsize\"] = (9, 4)\n",
    "plt.rcParams[\"axes.grid\"] = True\n",
    "plt.rcParams[\"grid.alpha\"] = 0.3\n",
    "\n",
    "def todo(hint):\n",
    "    '''Placeholder for a line you have to write.'''\n",
    "    raise NotImplementedError(f\"TODO: {hint}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d47a2616",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "> **A probability is a long-run relative frequency** (law of large numbers, [Probability](fundamentals-Probability.html#3.-The-law-of-large-numbers)).\n",
    ">\n",
    "> So every rule of the pen-and-paper exercises can be checked by simulating many repetitions and counting."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eacaec1d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Exercise 1 — The law of large numbers\n",
    "\n",
    "`running_mean` returns, for every $n$, the mean of the first $n$ values *([Probability](fundamentals-Probability.html#3.-The-law-of-large-numbers))*.\n",
    "\n",
    "**Task.** Fill in the `TODO` line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16651d03",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def running_mean(x):\n",
    "    '''Array whose n-th entry (0-based) is the mean of x[0], ..., x[n].'''\n",
    "    n = np.arange(1, len(x) + 1)\n",
    "    # TODO: cumulative sum divided by the number of values so far\n",
    "    return todo(\"np.cumsum(x) / n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "934e4b98",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "rng = np.random.default_rng(0)\n",
    "faces = rng.integers(1, 7, size=100_000)\n",
    "rm = running_mean(faces)\n",
    "fig, ax = plt.subplots(figsize=(9, 3))\n",
    "ax.plot(np.arange(1, len(rm) + 1), rm); ax.axhline(3.5, color=\"k\", ls=\":\")\n",
    "ax.set_xscale(\"log\"); ax.set_xlabel(\"n\"); ax.set_title(\"running mean of die rolls\"); plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a7ea9506",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# --- self-check Exercise 1 ---\n",
    "assert np.allclose(running_mean(np.array([2, 4, 6])), [2, 3, 4])\n",
    "assert abs(rm[-1] - 3.5) < 0.02, \"after 100 000 rolls the mean should be close to 3.5\"\n",
    "print(\"Exercise 1 ok\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d722117a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "**Questions — Exercise 1**\n",
    "\n",
    "1. Read the plot: roughly how far from $3.5$ is the running mean after $100$ rolls, and after\n",
    "   $10\\,000$? How does that compare with pen-and-paper Exercise 4, Task 3?\n",
    "2. Does the law of large numbers say the running mean can never move away from $3.5$ again?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7738415a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Your answer — Exercise 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "519ee618",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Exercise 2 — Conditional probabilities by counting\n",
    "\n",
    "`simulate_screening` draws a population: who is ill, and who tests positive. From these two\n",
    "arrays, `rates` estimates three conditional probabilities **by counting** *([Probability](fundamentals-Probability.html#5.-Conditional-probability-and-independence))*.\n",
    "\n",
    "**Task.** Fill in the two `TODO` lines: $p(\\text{pos} \\mid \\text{healthy})$ and $p(\\text{ill} \\mid \\text{pos})$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "583326b6",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def simulate_screening(n, prevalence, p_pos_ill, p_pos_healthy, rng):\n",
    "    '''Boolean arrays (ill, pos) for n people.'''\n",
    "    ill = rng.random(n) < prevalence\n",
    "    pos = np.where(ill, rng.random(n) < p_pos_ill, rng.random(n) < p_pos_healthy)\n",
    "    return ill, pos\n",
    "\n",
    "def rates(ill, pos):\n",
    "    '''(p(pos | ill), p(pos | healthy), p(ill | pos)) estimated by counting.'''\n",
    "    pos_ill = pos[ill].mean()                     # restrict to the ill, count positives\n",
    "    # TODO: p(pos | healthy)\n",
    "    pos_healthy = todo(\"mean of pos among the healthy\")\n",
    "    # TODO: p(ill | pos)\n",
    "    ill_pos = todo(\"mean of ill among the positive tests\")\n",
    "    return pos_ill, pos_healthy, ill_pos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e987c096",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "ill, pos = simulate_screening(1_000_000, 0.2, 0.95, 0.02, np.random.default_rng(1))\n",
    "pos_ill, pos_healthy, ill_pos = rates(ill, pos)\n",
    "print(f\"p(pos | ill) {pos_ill:.3f}   p(pos | healthy) {pos_healthy:.3f}   p(ill | pos) {ill_pos:.3f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cf4d7a1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# --- self-check Exercise 2 ---\n",
    "assert abs(pos_ill - 0.95) < 0.005 and abs(pos_healthy - 0.02) < 0.002\n",
    "assert abs(ill_pos - 0.19 / 0.206) < 0.005, \"p(ill | pos) should match pen-and-paper Exercise 3, Task 1\"\n",
    "print(\"Exercise 2 ok\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc0fe462",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "**Questions — Exercise 2**\n",
    "\n",
    "1. Which line of `rates` is \"conditioning restricts the sample space\"?\n",
    "2. Why is `pos[ill].mean()` not the same as `ill[pos].mean()`?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60e64ac2",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Your answer — Exercise 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b213e92e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Exercise 3 — Bayes' rule against the simulation\n",
    "\n",
    "**Task.** Fill in `ill_given_pos`: the denominator $p(\\text{pos})$ by marginalisation, then Bayes'\n",
    "rule *([Probability](fundamentals-Probability.html#6.-Bayes'-rule))*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25235e4e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def ill_given_pos(prevalence, p_pos_ill, p_pos_healthy):\n",
    "    '''p(ill | pos) by Bayes' rule.'''\n",
    "    # TODO: p(pos) = sum over ill and healthy\n",
    "    p_pos = todo(\"p_pos_ill * prevalence + p_pos_healthy * (1 - prevalence)\")\n",
    "    # TODO: Bayes' rule\n",
    "    return todo(\"p_pos_ill * prevalence / p_pos\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "737ebc4f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "sim_rng = np.random.default_rng(2)\n",
    "prevs = [0.5, 0.2, 0.05, 0.01, 0.002]\n",
    "for prev in prevs:\n",
    "    _, _, sim = rates(*simulate_screening(2_000_000, prev, 0.95, 0.02, sim_rng))\n",
    "    print(f\"prevalence {prev:5}:  Bayes {ill_given_pos(prev, 0.95, 0.02):.3f}   \"\n",
    "          f\"simulated {sim:.3f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d12d73b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# --- self-check Exercise 3 ---\n",
    "assert np.isclose(ill_given_pos(0.01, 0.9, 0.05), 0.009 / 0.0585)\n",
    "assert np.isclose(ill_given_pos(0.002, 0.95, 0.02), 0.0019 / 0.02186, rtol=1e-3)\n",
    "print(\"Exercise 3 ok\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df6bca00",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "**Questions — Exercise 3**\n",
    "\n",
    "1. At which prevalence is $p(\\text{ill} \\mid \\text{pos}) \\approx 0.5$ — half the positive tests belong to ill\n",
    "   patients?\n",
    "2. The simulation for prevalence $0.002$ is noisier than the others. Why? *(How many ill\n",
    "   people are among the positives?)*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3ae3354",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Your answer — Exercise 3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4d45cbe",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Exercise 4 — The binomial distribution\n",
    "\n",
    "**Task.** Fill in `binom_pmf` with `math.comb` *([Probability](fundamentals-Probability.html#10.-The-binomial-distribution))*; the code then compares it with\n",
    "simulated sums of Bernoulli trials."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1195a599",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def binom_pmf(k, n, theta):\n",
    "    '''p(K = k) for K ~ Binomial(n, theta).'''\n",
    "    # TODO: C(n, k) theta^k (1 - theta)^(n - k)\n",
    "    return todo(\"math.comb(n, k) * theta**k * (1 - theta)**(n - k)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd4186c5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "n_b, theta_b = 20, 0.3\n",
    "K = (np.random.default_rng(3).random((200_000, n_b)) < theta_b).sum(axis=1)\n",
    "pmf = np.array([binom_pmf(k, n_b, theta_b) for k in range(n_b + 1)])\n",
    "fig, ax = plt.subplots(figsize=(9, 3))\n",
    "ax.hist(K, bins=np.arange(-0.5, n_b + 1), density=True, color=\"steelblue\", label=\"simulated\")\n",
    "ax.plot(range(n_b + 1), pmf, \"ko-\", ms=3, label=\"pmf\"); ax.legend(); plt.show()\n",
    "print(f\"mean {K.mean():.3f}  var {K.var():.3f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "251ae728",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# --- self-check Exercise 4 ---\n",
    "assert np.isclose(pmf.sum(), 1.0), \"a pmf sums to 1\"\n",
    "assert np.isclose(binom_pmf(2, 4, 0.5), 6 / 16)\n",
    "assert np.isclose(K.mean(), n_b * theta_b, rtol=0.01)\n",
    "assert np.isclose(K.var(), n_b * theta_b * (1 - theta_b), rtol=0.02)\n",
    "print(\"Exercise 4 ok\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb68b066",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "**Questions — Exercise 4**\n",
    "\n",
    "1. Compute the mean and variance from `pmf` alone (two lines of `numpy`), and compare with\n",
    "   $n\\theta$ and $n\\theta(1-\\theta)$.\n",
    "2. Why did the binomial's mean and variance need no new calculation?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b59cb291",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Your answer — Exercise 4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22c6f96b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Exercise 5 — The Gaussian density\n",
    "\n",
    "**Task.** Fill in `gauss_pdf` *([Probability](fundamentals-Probability.html#11.-Continuous-variables-and-the-Gaussian))*. The code checks that it integrates to 1 and\n",
    "measures the probability of $\\mu \\pm 1.96\\,\\sigma$ by a Riemann sum."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2565c439",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def gauss_pdf(x, mu, sigma):\n",
    "    '''Density of N(mu, sigma^2) at x.'''\n",
    "    # TODO: exp(-(x - mu)^2 / (2 sigma^2)) / sqrt(2 pi sigma^2)\n",
    "    return todo(\"the Gaussian density\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fad42ffd",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "mu_g, sigma_g = 2.0, 0.1\n",
    "x = np.linspace(mu_g - 8 * sigma_g, mu_g + 8 * sigma_g, 200_001)\n",
    "dx = x[1] - x[0]\n",
    "dens = gauss_pdf(x, mu_g, sigma_g)\n",
    "inside = np.abs(x - mu_g) <= 1.96 * sigma_g\n",
    "print(f\"max density {dens.max():.3f}   integral {dens.sum() * dx:.5f}   \"\n",
    "      f\"p(mu ± 1.96 sigma) = {dens[inside].sum() * dx:.4f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19b4b666",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# --- self-check Exercise 5 ---\n",
    "assert np.isclose(gauss_pdf(0.0, 0.0, 1.0), 1 / np.sqrt(2 * np.pi))\n",
    "assert np.isclose(dens.sum() * dx, 1.0, atol=1e-4), \"a density integrates to 1\"\n",
    "assert np.isclose(dens[inside].sum() * dx, 0.95, atol=1e-3)\n",
    "print(\"Exercise 5 ok\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1226217",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "**Questions — Exercise 5**\n",
    "\n",
    "1. The maximum density is about $4$. Is that a contradiction to \"probabilities are at most 1\"?\n",
    "2. What is $p(X = 2.0)$ exactly, for this continuous $X$?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e528e500",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Your answer — Exercise 5"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d533d3c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "## Exercise 6 — The variance of a sum of dependent variables\n",
    "\n",
    "`correlated_pair` draws two Gaussian variables with variance 1 and correlation `rho`.\n",
    "\n",
    "**Task.** Fill in `var_sum_formula`: $\\operatorname{var}(X + Y)$ from the two variances and the covariance\n",
    "*([Probability](fundamentals-Probability.html#9.-Variance-and-covariance))*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "435e95a1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def correlated_pair(n, rho, rng):\n",
    "    x = rng.normal(0, 1, n)\n",
    "    y = rho * x + np.sqrt(1 - rho**2) * rng.normal(0, 1, n)\n",
    "    return x, y\n",
    "\n",
    "def var_sum_formula(x, y):\n",
    "    cov = np.mean(x * y) - x.mean() * y.mean()\n",
    "    # TODO: var(x) + var(y) + 2 cov\n",
    "    return todo(\"np.var(x) + np.var(y) + 2 * cov\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a4ae511",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "pair_rng = np.random.default_rng(4)\n",
    "for rho in [-0.9, 0.0, 0.5, 0.9]:\n",
    "    x, y = correlated_pair(500_000, rho, pair_rng)\n",
    "    print(f\"rho = {rho:+.1f}:  var(x + y) = {np.var(x + y):.3f}   formula {var_sum_formula(x, y):.3f}\"\n",
    "          f\"   var(x - y) = {np.var(x - y):.3f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9fb5fac4",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# --- self-check Exercise 6 ---\n",
    "x, y = correlated_pair(500_000, 0.9, np.random.default_rng(5))\n",
    "assert np.isclose(var_sum_formula(x, y), np.var(x + y)), \"the identity holds exactly for the sample\"\n",
    "assert np.var(x - y) < 0.3, \"strongly correlated: the difference should vary little\"\n",
    "print(\"Exercise 6 ok\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06495a19",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "**Questions — Exercise 6**\n",
    "\n",
    "1. For $\\rho = 0.9$, $\\operatorname{var}(x - y)$ is about $0.2$, although each variable has variance 1. Explain\n",
    "   with the formula for $\\operatorname{var}(X - Y)$.\n",
    "2. Two models are evaluated on the same test set. What does this effect mean for the\n",
    "   difference of their accuracies? *([Uncertainty](fundamentals-Uncertainty.html#6.-Comparing-two-estimates) puts it to use.)*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db240259",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Your answer — Exercise 6"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98d95e61",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Literature\n",
    "\n",
    "- L. Wasserman: *All of Statistics*, Springer 2004 — ch. 1–3"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  },
  "rise": {
   "scroll": true,
   "theme": "white",
   "transition": "none"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}