{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ac434d8a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Lab\n",
    "\n",
    "**Machine Learning** · Fundamentals · *Uncertainty*\n",
    "\n",
    "The tools for error bars, built and checked by simulation: the coverage of a confidence interval,\n",
    "the bootstrap, a paired comparison, maximum likelihood numerically."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0f38adc",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## How to work through this\n",
    "\n",
    "The code is given — read it line by line. Only the lines marked `# TODO` are yours: they are the\n",
    "lines that carry the idea. Until you replace it, each `todo(...)` stops with an error that says what\n",
    "is missing. The self-checks test your `TODO` lines.\n",
    "\n",
    "After each block, answer the **questions** in the cell below it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f08c565",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2026-11-12T10:19:08.783733Z",
     "iopub.status.busy": "2026-11-12T10:19:08.783634Z",
     "iopub.status.idle": "2026-11-12T10:19:09.094846Z",
     "shell.execute_reply": "2026-11-12T10:19:09.094250Z"
    },
    "slideshow": {
     "slide_type": "-"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline\n",
    "np.set_printoptions(precision=4, suppress=True)\n",
    "plt.rcParams[\"figure.figsize\"] = (9, 4)\n",
    "plt.rcParams[\"axes.grid\"] = True\n",
    "plt.rcParams[\"grid.alpha\"] = 0.3\n",
    "rng = np.random.default_rng(0)\n",
    "\n",
    "def todo(hint):\n",
    "    \"\"\"Placeholder for a line you have to write.\"\"\"\n",
    "    raise NotImplementedError(f\"TODO: {hint}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "359f42ff",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "This part rests on one idea:\n",
    "\n",
    "> **Standard errors and confidence intervals are statements about many data sets.**\n",
    ">\n",
    "> We have one data set — so we *simulate* many, and check that what one data set says\n",
    "> about its own uncertainty is right."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be5e30a1",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Exercise 1 — Does a 95 % interval cover 95 %?\n",
    "\n",
    "`ci_coverage` simulates `reps` test sets of $n$ examples, each classified correctly with\n",
    "probability $\\theta$, builds the interval $\\hat\\theta \\pm z\\,\\widehat{\\operatorname{se}}$ on each, and returns the share\n",
    "of intervals that contain the true $\\theta$ *([Uncertainty](fundamentals-Uncertainty.html#2.-Confidence-intervals))*.\n",
    "\n",
    "**Task.** Fill in the two `TODO` lines: the standard error and the \"covered\" test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b41e9384",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-12T10:19:09.097431Z",
     "iopub.status.busy": "2026-11-12T10:19:09.097271Z",
     "iopub.status.idle": "2026-11-12T10:19:09.100380Z",
     "shell.execute_reply": "2026-11-12T10:19:09.100187Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def ci_coverage(theta, n, reps, rng, z=1.96):\n",
    "    '''Share of the intervals theta_hat ± z * se_hat that contain theta.'''\n",
    "    k = rng.binomial(n, theta, size=reps)      # number correct in each test set\n",
    "    theta_hat = k / n\n",
    "    # TODO: the estimated standard error of every theta_hat (Uncertainty §4)\n",
    "    se_hat = todo(\"se_hat = sqrt(theta_hat (1 - theta_hat) / n)\")\n",
    "    # TODO: True where the interval contains theta\n",
    "    covered = todo(\"covered: |theta_hat - theta| <= z * se_hat\")\n",
    "    return covered.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3593c05",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-12T10:19:09.104609Z",
     "iopub.status.busy": "2026-11-12T10:19:09.104524Z",
     "iopub.status.idle": "2026-11-12T10:19:09.123793Z",
     "shell.execute_reply": "2026-11-12T10:19:09.123395Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "cov_rng = np.random.default_rng(1)\n",
    "for theta, n_ in [(0.88, 100), (0.88, 1000), (0.95, 10)]:\n",
    "    print(f\"theta = {theta}, n = {n_:4}:  coverage = \"\n",
    "          f\"{ci_coverage(theta, n_, 100_000, cov_rng):.3f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3222f155",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-12T10:19:09.125002Z",
     "iopub.status.busy": "2026-11-12T10:19:09.124898Z",
     "iopub.status.idle": "2026-11-12T10:19:09.142595Z",
     "shell.execute_reply": "2026-11-12T10:19:09.142313Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# --- self-check Exercise 1 ---\n",
    "c1 = ci_coverage(0.88, 1000, 100_000, np.random.default_rng(2))\n",
    "c2 = ci_coverage(0.95, 10, 100_000, np.random.default_rng(2))\n",
    "c3 = ci_coverage(0.88, 1000, 100_000, np.random.default_rng(2), z=1.0)\n",
    "assert 0.94 < c1 < 0.96, \"for large n the 95 % interval should cover about 95 %\"\n",
    "assert c2 < 0.6, \"for n = 10, theta = 0.95 the normal approximation should fail badly\"\n",
    "assert 0.66 < c3 < 0.70, \"± 1 se covers about 68 %\"\n",
    "print(\"Exercise 1 ok\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "202b7dd6",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Questions — Exercise 1**\n",
    "\n",
    "1. For $\\theta = 0.95$, $n = 10$ the coverage is far below 95 %. What happens in the\n",
    "   most likely test set, the one with $k = 10$? *(Compute $p(k = 10)$.)*\n",
    "2. Which coverage would you expect with `z=1.0`? Check it.\n",
    "3. A report says: \"the accuracy lies in $[0.84, 0.92]$ with probability 95 %\". What is\n",
    "   wrong with this sentence, and how should it be phrased?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b467e87f",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Your answer — Exercise 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f00f05f0",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Exercise 2 — The bootstrap\n",
    "\n",
    "`bootstrap` draws `B` resamples of the rows of `data` — with replacement — and evaluates\n",
    "`stat` on each *([Uncertainty](fundamentals-Uncertainty.html#5.-The-bootstrap))*.\n",
    "\n",
    "**Task.** Fill in the two `TODO` lines: draw the indices, and evaluate the statistic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c902fbd0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-12T10:19:09.143770Z",
     "iopub.status.busy": "2026-11-12T10:19:09.143689Z",
     "iopub.status.idle": "2026-11-12T10:19:09.145992Z",
     "shell.execute_reply": "2026-11-12T10:19:09.145692Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def bootstrap(stat, data, B, rng):\n",
    "    '''B bootstrap replicates of stat(data); data has one example per row.'''\n",
    "    n = len(data)\n",
    "    reps = np.empty(B)\n",
    "    for b in range(B):\n",
    "        # TODO: n indices drawn uniformly WITH replacement from 0 .. n-1\n",
    "        idx = todo(\"idx = rng.integers(0, n, size=n)\")\n",
    "        # TODO: the statistic on the resampled rows\n",
    "        reps[b] = todo(\"reps[b] = stat(data[idx])\")\n",
    "    return reps"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5dc30ff",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The test set of pen-and-paper Exercise 1 as an array of $(y, \\hat y)$ rows, and $F_1$ computed from it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05891a3a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-12T10:19:09.150237Z",
     "iopub.status.busy": "2026-11-12T10:19:09.150161Z",
     "iopub.status.idle": "2026-11-12T10:19:09.152738Z",
     "shell.execute_reply": "2026-11-12T10:19:09.152550Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "pairs = np.array([[1, 1]] * 45 + [[1, 0]] * 15 + [[0, 1]] * 5 + [[0, 0]] * 135)\n",
    "\n",
    "def f1(p):\n",
    "    tp = np.sum((p[:, 0] == 1) & (p[:, 1] == 1))\n",
    "    fp = np.sum((p[:, 0] == 0) & (p[:, 1] == 1))\n",
    "    fn = np.sum((p[:, 0] == 1) & (p[:, 1] == 0))\n",
    "    return 2 * tp / (2 * tp + fp + fn)\n",
    "\n",
    "def accuracy(p):\n",
    "    return np.mean(p[:, 0] == p[:, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "985499da",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-12T10:19:09.153659Z",
     "iopub.status.busy": "2026-11-12T10:19:09.153582Z",
     "iopub.status.idle": "2026-11-12T10:19:09.286907Z",
     "shell.execute_reply": "2026-11-12T10:19:09.286596Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "boot_rng = np.random.default_rng(3)\n",
    "reps_acc = bootstrap(accuracy, pairs, 5000, boot_rng)\n",
    "reps_f1 = bootstrap(f1, pairs, 5000, boot_rng)\n",
    "print(f\"accuracy = {accuracy(pairs):.2f} ± {1.96 * reps_acc.std():.2f}   \"\n",
    "      f\"(formula: ± {1.96 * np.sqrt(0.9 * 0.1 / 200):.2f})\")\n",
    "print(f\"F1       = {f1(pairs):.2f} ± {1.96 * reps_f1.std():.2f}   \"\n",
    "      f\"percentile [{np.quantile(reps_f1, 0.025):.2f}, {np.quantile(reps_f1, 0.975):.2f}]\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a7dd897",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-12T10:19:09.288099Z",
     "iopub.status.busy": "2026-11-12T10:19:09.288010Z",
     "iopub.status.idle": "2026-11-12T10:19:09.290833Z",
     "shell.execute_reply": "2026-11-12T10:19:09.290564Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# --- self-check Exercise 2 ---\n",
    "assert reps_acc.shape == (5000,)\n",
    "assert np.isclose(reps_acc.std(), np.sqrt(0.9 * 0.1 / 200), rtol=0.1), \\\n",
    "    \"bootstrap se of the accuracy should match the formula of Uncertainty §4\"\n",
    "assert 0.02 < reps_f1.std() < 0.06, \"the bootstrap se of F1 looks implausible\"\n",
    "assert len(np.unique(reps_acc)) > 10, \"are you sampling WITH replacement?\"\n",
    "print(\"Exercise 2 ok\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "408f46c4",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Questions — Exercise 2**\n",
    "\n",
    "1. What would `bootstrap` return if the indices were drawn **without** replacement?\n",
    "2. Why are whole rows $(y, \\hat y)$ resampled, and not the four counts of the confusion\n",
    "   matrix independently of each other?\n",
    "3. The test set was used to choose among 20 models, and the best one is reported with a\n",
    "   bootstrap error bar. What does the error bar capture, and what does it miss?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edf51fd7",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Your answer — Exercise 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5568a42",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Exercise 3 — Comparing two classifiers on the same test set\n",
    "\n",
    "`two_classifiers` simulates the correctness $c_A^{(i)}, c_B^{(i)} \\in \\{0,1\\}$ of two models on the\n",
    "same $n$ examples. The models share the difficulty of each example, so they mostly agree\n",
    "*([Uncertainty](fundamentals-Uncertainty.html#6.-Comparing-two-estimates))*.\n",
    "\n",
    "**Task.** Fill in the two `TODO` lines: the standard error of the independent Wald test,\n",
    "and that of the paired test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f85b3af0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-12T10:19:09.291842Z",
     "iopub.status.busy": "2026-11-12T10:19:09.291756Z",
     "iopub.status.idle": "2026-11-12T10:19:09.293962Z",
     "shell.execute_reply": "2026-11-12T10:19:09.293494Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def two_classifiers(n, acc_a, acc_b, rng, redraw=0.2):\n",
    "    '''Correctness of A and B on the same n examples, sharing example difficulty.'''\n",
    "    u = rng.random(n)\n",
    "    u_b = np.where(rng.random(n) < redraw, rng.random(n), u)\n",
    "    return (u < acc_a).astype(float), (u_b < acc_b).astype(float)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c716df8a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-12T10:19:09.295107Z",
     "iopub.status.busy": "2026-11-12T10:19:09.295009Z",
     "iopub.status.idle": "2026-11-12T10:19:09.297293Z",
     "shell.execute_reply": "2026-11-12T10:19:09.296953Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def z_statistics(ca, cb):\n",
    "    '''Wald statistic for acc(A) - acc(B): (independent, paired).'''\n",
    "    n, a, b = len(ca), ca.mean(), cb.mean()\n",
    "    # TODO: se of a - b if the two accuracies were independent\n",
    "    se_ind = todo(\"se_ind = sqrt(a(1-a)/n + b(1-b)/n)\")\n",
    "    d = ca - cb\n",
    "    # TODO: se of the mean of the per-example differences d\n",
    "    se_pair = todo(\"se_pair = d.std(ddof=1) / sqrt(n)\")\n",
    "    return (a - b) / se_ind, d.mean() / se_pair"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bce2f549",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-12T10:19:09.303782Z",
     "iopub.status.busy": "2026-11-12T10:19:09.303696Z",
     "iopub.status.idle": "2026-11-12T10:19:09.443770Z",
     "shell.execute_reply": "2026-11-12T10:19:09.443448Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def rejection_rates(acc_a, acc_b, n=500, trials=2000, seed=4):\n",
    "    '''Share of simulated test sets with Z > 1.96: (independent, paired).'''\n",
    "    rng_ = np.random.default_rng(seed)\n",
    "    Z = np.array([z_statistics(*two_classifiers(n, acc_a, acc_b, rng_))\n",
    "                  for _ in range(trials)])\n",
    "    return (Z > 1.96).mean(axis=0)\n",
    "\n",
    "for acc_b in [0.86, 0.84, 0.82]:\n",
    "    ind, pair = rejection_rates(0.86, acc_b)\n",
    "    print(f\"acc A = 0.86, acc B = {acc_b}:  independent {ind:.3f}   paired {pair:.3f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1302cf7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-12T10:19:09.445113Z",
     "iopub.status.busy": "2026-11-12T10:19:09.445014Z",
     "iopub.status.idle": "2026-11-12T10:19:09.534400Z",
     "shell.execute_reply": "2026-11-12T10:19:09.533948Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# --- self-check Exercise 3 ---\n",
    "ind0, pair0 = rejection_rates(0.86, 0.86)\n",
    "ind4, pair4 = rejection_rates(0.86, 0.82)\n",
    "assert pair0 < 0.05, \"with equal accuracies the paired test should rarely claim a difference\"\n",
    "assert pair4 > ind4 + 0.3, \"the paired test should detect the real difference far more often\"\n",
    "print(\"Exercise 3 ok\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c4961aa",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Questions — Exercise 3**\n",
    "\n",
    "1. In the first row both models are equally good. What should the share of \"A is\n",
    "   better\" verdicts be, ideally? Which test comes closer, and why is the independent one\n",
    "   *too cautious* rather than too eager?\n",
    "2. How does the paired test's advantage depend on `redraw` — the share of examples on\n",
    "   which B's difficulty differs from A's? Predict, then try `redraw=1.0`.\n",
    "3. A paper reports model A: $0.86 \\pm 0.03$, model B: $0.82 \\pm 0.03$, on the same test set,\n",
    "   and concludes \"no significant difference, the bars overlap\". What would you ask for?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64369c34",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Your answer — Exercise 3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "625e81da",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Exercise 4 — The maximum-likelihood estimate, numerically\n",
    "\n",
    "You derived $\\hat\\theta_{ML} = k/n$ in pen-and-paper Exercise 3. Here you find it **without** the derivation — by\n",
    "minimising the NLL with gradient descent, using a finite-difference derivative — and read the\n",
    "error bar off the curvature *([Uncertainty](fundamentals-Uncertainty.html#3.-Maximum-likelihood-for-the-binomial))*.\n",
    "\n",
    "**Task.** Fill in the two `TODO` lines: the NLL, and the gradient-descent update."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f58bccab",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-12T10:19:09.535826Z",
     "iopub.status.busy": "2026-11-12T10:19:09.535728Z",
     "iopub.status.idle": "2026-11-12T10:19:09.538074Z",
     "shell.execute_reply": "2026-11-12T10:19:09.537866Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def nll(theta, k, n):\n",
    "    '''Binomial negative log-likelihood, without the constant log C(n, k).'''\n",
    "    # TODO: -( k log(theta) + (n - k) log(1 - theta) )\n",
    "    return todo(\"the binomial NLL\")\n",
    "\n",
    "def d_nll(theta, k, n, eps=1e-6):\n",
    "    '''Derivative of the NLL by central differences (the gradient check of L-LinReg).'''\n",
    "    return (nll(theta + eps, k, n) - nll(theta - eps, k, n)) / (2 * eps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "160061d4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-12T10:19:09.539035Z",
     "iopub.status.busy": "2026-11-12T10:19:09.538954Z",
     "iopub.status.idle": "2026-11-12T10:19:09.541129Z",
     "shell.execute_reply": "2026-11-12T10:19:09.540818Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def fit_theta(k, n, theta=0.5, alpha=0.05, n_iter=500):\n",
    "    '''Gradient descent on the MEAN NLL.'''\n",
    "    for _ in range(n_iter):\n",
    "        # TODO: one gradient-descent step on nll / n\n",
    "        theta = todo(\"theta = theta - alpha * d_nll(theta, k, n) / n\")\n",
    "    return theta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7bec4091",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-12T10:19:09.544685Z",
     "iopub.status.busy": "2026-11-12T10:19:09.544601Z",
     "iopub.status.idle": "2026-11-12T10:19:09.548263Z",
     "shell.execute_reply": "2026-11-12T10:19:09.548067Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "k_obs, n_obs = 45, 50                       # the precision of pen-and-paper Exercise 1\n",
    "theta_ml = fit_theta(k_obs, n_obs)\n",
    "eps = 1e-4\n",
    "curv = (nll(theta_ml + eps, k_obs, n_obs) - 2 * nll(theta_ml, k_obs, n_obs)\n",
    "        + nll(theta_ml - eps, k_obs, n_obs)) / eps**2\n",
    "print(f\"gradient descent: theta = {theta_ml:.4f}     k/n = {k_obs / n_obs:.4f}\")\n",
    "print(f\"se from curvature: {1 / np.sqrt(curv):.4f}   formula: \"\n",
    "      f\"{np.sqrt(0.9 * 0.1 / n_obs):.4f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49667389",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-12T10:19:09.549374Z",
     "iopub.status.busy": "2026-11-12T10:19:09.549287Z",
     "iopub.status.idle": "2026-11-12T10:19:09.551403Z",
     "shell.execute_reply": "2026-11-12T10:19:09.551198Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "y = np.r_[np.ones(k_obs), np.zeros(n_obs - k_obs)]\n",
    "ce = -np.mean(y * np.log(theta_ml) + (1 - y) * np.log(1 - theta_ml))\n",
    "print(f\"cross-entropy at theta_ml: {ce:.5f}    NLL / n: {nll(theta_ml, k_obs, n_obs) / n_obs:.5f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6096364",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-12T10:19:09.552356Z",
     "iopub.status.busy": "2026-11-12T10:19:09.552278Z",
     "iopub.status.idle": "2026-11-12T10:19:09.554675Z",
     "shell.execute_reply": "2026-11-12T10:19:09.554450Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# --- self-check Exercise 4 ---\n",
    "assert np.isclose(nll(0.9, 45, 50), -(45 * np.log(0.9) + 5 * np.log(0.1)))\n",
    "assert np.isclose(theta_ml, 0.9, atol=1e-4), \"gradient descent should land on k/n\"\n",
    "assert np.isclose(1 / np.sqrt(curv), np.sqrt(0.09 / 50), rtol=1e-3), \\\n",
    "    \"the inverse curvature should equal theta(1-theta)/n (pen-and-paper Exercise 3, Task 3)\"\n",
    "assert np.isclose(ce, nll(theta_ml, k_obs, n_obs) / n_obs), \"NLL / n is the cross-entropy\"\n",
    "print(\"Exercise 4 ok\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "417bf7a8",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Questions — Exercise 4**\n",
    "\n",
    "1. Why does the binomial coefficient not appear in `nll` — and why does that not change\n",
    "   `theta_ml`?\n",
    "2. The step is taken on the **mean** NLL, `d_nll / n`. What would happen with `alpha = 0.05`\n",
    "   on the summed NLL for $n = 50\\,000$? *(Hint: the stability bound of → L-LinReg,\n",
    "   $\\alpha < 2/\\lambda_{\\max}(H)$, with the curvature computed above.)*\n",
    "3. Repeat with $k = 450$, $n = 500$. How do `theta_ml` and the curvature change, and what\n",
    "   does that mean for the error bar?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50d35d2d",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Your answer — Exercise 4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b6feac0",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d6d7a63",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "You computed, by hand and by simulation, what lies behind every error bar:\n",
    "\n",
    "| quantity | formula | exercises · lab |\n",
    "|---|---|---|\n",
    "| se of a proportion | $\\sqrt{\\hat\\theta(1-\\hat\\theta)/n}$, with the metric's own $n$ | 1, lab 1 |\n",
    "| se without a formula | bootstrap over test examples | lab 2 |\n",
    "| comparing two models | paired $Z = \\bar d / (\\operatorname{std}(d)/\\sqrt n)$ | 2, lab 3 |\n",
    "| where $k/n$ comes from | maximum likelihood = minimum cross-entropy | 3, lab 4 |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0554e9dc",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Literature\n",
    "\n",
    "- L. Wasserman: *All of Statistics*, Springer 2004 — ch. 6, ch. 8, §9.7, §10.1\n",
    "- I. Goodfellow, Y. Bengio, A. Courville: *Deep Learning*, MIT Press 2016 — §5.4, §5.5"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  },
  "rise": {
   "scroll": true,
   "theme": "white",
   "transition": "none"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}