{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f6ecdb00",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Lab\n",
    "\n",
    "**Machine Learning** · Fundamentals · *Estimators*\n",
    "\n",
    "The results of the pen-and-paper exercises (`fundamentals-Estimators-exercises.ipynb`), checked by\n",
    "simulation: estimators over many data sets."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1065f555",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## How to work through this\n",
    "\n",
    "The code is given — read it line by line. Only the lines marked `# TODO` are yours: they are the\n",
    "lines that carry the idea. Until you replace it, each `todo(...)` stops with an error that says what\n",
    "is missing. The self-checks test your `TODO` lines.\n",
    "\n",
    "After each block, answer the **questions** in the cell below it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e2e823b",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2026-11-12T10:19:03.228332Z",
     "iopub.status.busy": "2026-11-12T10:19:03.228116Z",
     "iopub.status.idle": "2026-11-12T10:19:03.554412Z",
     "shell.execute_reply": "2026-11-12T10:19:03.553849Z"
    },
    "slideshow": {
     "slide_type": "-"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline\n",
    "np.set_printoptions(precision=4, suppress=True)\n",
    "plt.rcParams[\"figure.figsize\"] = (9, 4)\n",
    "plt.rcParams[\"axes.grid\"] = True\n",
    "plt.rcParams[\"grid.alpha\"] = 0.3\n",
    "rng = np.random.default_rng(0)\n",
    "\n",
    "def todo(hint):\n",
    "    \"\"\"Placeholder for a line you have to write.\"\"\"\n",
    "    raise NotImplementedError(f\"TODO: {hint}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b7bd520",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "This part rests on one idea:\n",
    "\n",
    "> **Bias and variance are statements about many data sets.**\n",
    ">\n",
    "> We have one data set — so we *simulate* many, and check what you proved in the pen-and-paper exercises."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1547fe9e",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Exercise 1 — Bias and variance of estimators\n",
    "\n",
    "`simulate_estimators` draws `reps` data sets of size `n` from $\\mathcal{N}(\\mu, \\sigma^2)$ and\n",
    "evaluates the three estimators of *([Estimators](fundamentals-Estimators.html#2.-Bias-and-variance))* on each: $\\hat\\mu$, $\\hat\\sigma_b^2$,\n",
    "$\\hat\\sigma_u^2$. One row of `X` is one data set; every reduction runs along `axis=1`.\n",
    "\n",
    "**Task.** Fill in the two `TODO` lines: the biased and the unbiased sample variance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61bbd9a7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-12T10:19:03.556585Z",
     "iopub.status.busy": "2026-11-12T10:19:03.556422Z",
     "iopub.status.idle": "2026-11-12T10:19:03.559314Z",
     "shell.execute_reply": "2026-11-12T10:19:03.559003Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def simulate_estimators(reps, n, mu, sigma, rng):\n",
    "    \"\"\"Draw `reps` data sets of size n and evaluate three estimators on each.\n",
    "\n",
    "    Returns: (mu_hat, var_b, var_u), each an array of shape (reps,)\n",
    "    \"\"\"\n",
    "    X = rng.normal(mu, sigma, size=(reps, n))     # one data set per row\n",
    "    mu_hat = X.mean(axis=1)                       # sample mean of every data set\n",
    "    dev2 = (X - mu_hat[:, None]) ** 2             # squared deviations from the FITTED mean\n",
    "    # TODO: the biased sample variance (pen-and-paper Exercise 2) — divide the row sums of dev2 by n\n",
    "    var_b = todo(\"var_b: divide the row sums of dev2 by n\")\n",
    "    # TODO: the unbiased sample variance — divide by n - 1\n",
    "    var_u = todo(\"var_u: divide the row sums of dev2 by n - 1\")\n",
    "    return mu_hat, var_b, var_u"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f2b8722",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2026-11-12T10:19:03.563159Z",
     "iopub.status.busy": "2026-11-12T10:19:03.562973Z",
     "iopub.status.idle": "2026-11-12T10:19:03.581276Z",
     "shell.execute_reply": "2026-11-12T10:19:03.580674Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "mu_true, sigma_true, n, reps = 3.0, 2.0, 5, 100_000\n",
    "mu_hat, var_b, var_u = simulate_estimators(reps, n, mu_true, sigma_true,\n",
    "                                           np.random.default_rng(0))\n",
    "print(\"shapes:\", mu_hat.shape, var_b.shape, var_u.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f49f6b9",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2026-11-12T10:19:03.582950Z",
     "iopub.status.busy": "2026-11-12T10:19:03.582838Z",
     "iopub.status.idle": "2026-11-12T10:19:03.586580Z",
     "shell.execute_reply": "2026-11-12T10:19:03.586306Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# --- self-check Exercise 1 ---\n",
    "assert mu_hat.shape == (reps,), \"one value per simulated data set\"\n",
    "assert np.isclose(mu_hat.mean(), mu_true, atol=0.02), \"mu_hat should be unbiased (pen-and-paper Exercise 1)\"\n",
    "assert np.isclose(mu_hat.var(), sigma_true**2 / n, rtol=0.05), \"var(mu_hat) = sigma^2/n (pen-and-paper Exercise 1)\"\n",
    "assert np.isclose(var_b.mean(), (n - 1) / n * sigma_true**2, rtol=0.02), \\\n",
    "    \"E[sigma_b^2] = (n-1)/n * sigma^2 (pen-and-paper Exercise 2)\"\n",
    "assert np.isclose(var_u.mean(), sigma_true**2, rtol=0.02), \"sigma_u^2 should be unbiased (pen-and-paper Exercise 2)\"\n",
    "print(\"Exercise 1 ok — your simulation reproduces the results you proved in pen-and-paper Exercise 1 and pen-and-paper Exercise 2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e8f349a",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Questions — Exercise 1**\n",
    "\n",
    "1. Why does `X` have one row per data set, and why do all reductions use `axis=1`?\n",
    "   What would `X.mean(axis=0)` compute instead?\n",
    "2. Which line of the code is \"$\\bar X$ is fitted to the same data\" from pen-and-paper Exercise 2, Task 2?\n",
    "3. The run above uses $n = 5$. Which bias of `var_b` do you expect for $n = 100$ (with\n",
    "   $\\sigma = 2$)? Why does the choice between the two estimators matter less for large $n$?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b51f64a",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Your answer — Exercise 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8910a1f0",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Exercise 2 — The mse identity, numerically\n",
    "\n",
    "`bias_variance_mse` returns $\\operatorname{bias}$, $\\operatorname{var}$ and $\\operatorname{mse}$\n",
    "of a sample of estimates; the printout then checks\n",
    "$\\operatorname{mse} = \\operatorname{bias}^2 + \\operatorname{var}$ *(pen-and-paper Exercise 3)*.\n",
    "\n",
    "**Task.** Fill in the variance. Careful: the spread around *which* value?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cebf633d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-11-12T10:19:03.587681Z",
     "iopub.status.busy": "2026-11-12T10:19:03.587592Z",
     "iopub.status.idle": "2026-11-12T10:19:03.589434Z",
     "shell.execute_reply": "2026-11-12T10:19:03.589255Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def bias_variance_mse(estimates, true_value):\n",
    "    \"\"\"Empirical bias, variance and mse of an estimator.\n",
    "\n",
    "    Args:\n",
    "        estimates:  array of estimator values, one per simulated data set\n",
    "        true_value: the true parameter\n",
    "    Returns:\n",
    "        (bias, variance, mse)\n",
    "    \"\"\"\n",
    "    estimates = np.asarray(estimates)\n",
    "    bias = estimates.mean() - true_value           # E[theta_hat] - theta\n",
    "    mse = np.mean((estimates - true_value) ** 2)   # E[(theta_hat - theta)^2]\n",
    "    # TODO: the variance of the estimator\n",
    "    variance = todo(\"variance: mean squared deviation of the estimates from their own mean\")\n",
    "    return bias, variance, mse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33a4f014",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2026-11-12T10:19:03.593448Z",
     "iopub.status.busy": "2026-11-12T10:19:03.593368Z",
     "iopub.status.idle": "2026-11-12T10:19:03.596143Z",
     "shell.execute_reply": "2026-11-12T10:19:03.595884Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "for name, est, truth in [(\"mu_hat\", mu_hat, mu_true),\n",
    "                         (\"sigma_b^2\", var_b, sigma_true**2),\n",
    "                         (\"sigma_u^2\", var_u, sigma_true**2)]:\n",
    "    b, v, mse = bias_variance_mse(est, truth)\n",
    "    print(f\"{name:10} bias = {b:+8.4f}  bias^2 = {b**2:8.4f}  var = {v:8.4f}  \"\n",
    "          f\"sum = {b**2 + v:8.4f}  mse = {mse:8.4f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb3d93c3",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2026-11-12T10:19:03.597187Z",
     "iopub.status.busy": "2026-11-12T10:19:03.596993Z",
     "iopub.status.idle": "2026-11-12T10:19:03.600117Z",
     "shell.execute_reply": "2026-11-12T10:19:03.599828Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# --- self-check Exercise 2 ---\n",
    "for est, truth in [(mu_hat, mu_true), (var_b, sigma_true**2), (var_u, sigma_true**2)]:\n",
    "    b, v, mse = bias_variance_mse(est, truth)\n",
    "    assert np.isclose(b**2 + v, mse, rtol=1e-10), \"mse must equal bias^2 + variance (pen-and-paper Exercise 3)\"\n",
    "b_b, _, mse_b = bias_variance_mse(var_b, sigma_true**2)\n",
    "_, _, mse_u = bias_variance_mse(var_u, sigma_true**2)\n",
    "assert b_b < 0 and mse_b < mse_u, \"the biased estimator should have the LOWER mse here\"\n",
    "print(\"Exercise 2 ok — and the biased estimator wins on mse (pen-and-paper Exercise 3 follow-up)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff25fef4",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Questions — Exercise 2**\n",
    "\n",
    "1. `variance` and `mse` are both mean squared deviations. From which value does each one\n",
    "   measure the deviation?\n",
    "2. The self-check demands $\\operatorname{bias}^2 + \\operatorname{var} = \\operatorname{mse}$ to a\n",
    "   relative tolerance of $10^{-10}$ — far tighter than the simulation noise. Why does the\n",
    "   identity hold exactly for the simulated sample, not just approximately?\n",
    "   *(Hint: repeat pen-and-paper Exercise 3 with $\\mathbb{E}$ replaced by the average over the `reps` values.)*\n",
    "3. Read the printed table: which estimator of $\\sigma^2$ has the lower mse, and which of the\n",
    "   two terms decides it?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31d5c803",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Your answer — Exercise 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94d67da8",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd353963",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "You computed, by hand and by simulation, the quality of three estimators:\n",
    "\n",
    "| quantity | result | exercises · lab |\n",
    "|---|---|---|\n",
    "| sample mean $\\hat\\mu$ | unbiased, $\\operatorname{var}(\\hat\\mu) = \\sigma^2/n$ | 1, lab 1 |\n",
    "| sample variance | $\\mathbb{E}[\\hat\\sigma_b^2] = \\frac{n-1}{n}\\sigma^2$; $\\hat\\sigma_u^2$ is unbiased | 2, lab 1 |\n",
    "| bias, variance, mse | $\\operatorname{mse} = \\operatorname{bias}^2 + \\operatorname{var}$ | 3, lab 2 |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d0ba3e6",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Literature\n",
    "\n",
    "- L. Wasserman: *All of Statistics*, Springer 2004 — ch. 6\n",
    "- I. Goodfellow, Y. Bengio, A. Courville: *Deep Learning*, MIT Press 2016 — §5.4"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  },
  "rise": {
   "scroll": true,
   "theme": "white",
   "transition": "none"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}