Introduction to Machine Learning — Course

lecture + exercises + lab

Machine learning from first principles: every model is derived on paper and implemented from scratch in numpy before a framework takes over — from linear regression via the bias–variance trade-off and model evaluation to neural networks, language models, attention and Transformers. Automatic differentiation (PyTorch) arrives right before the neural networks.

Prerequisites: Mathematics (calculus, linear algebra, probability) and programming in Python.

Contents (preliminary syllabus)

Status: Alpha - Work in progress

Part I — Linear Models

Lecture Topic Slides Exercises Lab
1 L-LinReg Univariate Linear Regression and Gradient Descent
2 L-MultiReg Multivariate Linear Regression and Feature Scaling
3 L-LogReg Logistic Regression and the Cross-Entropy

Part II — Generalisation and Evaluation

Lecture Topic Slides Exercises Lab
4 L-BiasVar The Bias–Variance Trade-off lecture exercises
5 L-Eval Regularisation, Validation and Evaluation
6 L-Fair Fairness Metrics and the Impossibility Theorem

Part III — Neural Networks

Lecture Topic Slides Exercises Lab
7 L-Autodiff Automatic Differentiation
8 L-FFNN Feed-Forward Neural Networks
9 L-Train Training Deep Networks: Adam, Normalisation, Residuals

Part IV — Language Models

Lecture Topic Slides Exercises Lab
10 L-LM Language Modelling and Information Theory
11 L-Embed Embeddings and the Neural Language Model
12 L-Att Attention
13 L-Transformer Transformers

Extended Course (optional)

Lecture Topic Slides Exercises Lab
14 L-Classical Classical ML: Entropy, Decision Trees, Ensembles

Fundamentals

A fundamentals notebook recaps the mathematics the lectures need, with its own pen-and-paper exercises and lab.

Topic Content Used in Notes Exercises Lab
Probability sample space and axioms, random variables, joint, marginal and conditional distributions, Bayes' rule, conditional independence, expectation, variance and covariance, the binomial and the Gaussian L-BiasVar, L-Eval, L-Fair, L-LM notes exercises lab
Estimators statistic and estimator, bias, variance and mean squared error, sufficient statistics L-BiasVar notes exercises lab
Uncertainty standard error, 95 % confidence interval, maximum likelihood, error bars for proportions, the bootstrap, comparing two estimates L-Eval notes exercises lab
Linear Algebra vectors and matrices, dot and matrix product, the shapes a network layer needs L-MultiReg, L-FFNN

Literature & further reading

Machine learning

  • Abu-Mostafa, Y.; Magdon-Ismail, M.; Lin, H.-T. (2012). Learning from Data. AMLBook — generalisation, the bias–variance trade-off, validation (L-BiasVar, L-Eval).
  • Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer — linear models for regression and classification (L-LinReg, L-MultiReg, L-LogReg).
  • Hastie, T.; Tibshirani, R.; Friedman, J. (2009). The Elements of Statistical Learning (2nd ed.). Springer — model assessment and selection (L-BiasVar, L-Eval, L-Classical).
  • Goodfellow, I.; Bengio, Y.; Courville, A. (2016). Deep Learning. MIT Press — full text at deeplearningbook.org (L-Autodiff, L-FFNN, L-Train).

Probability & statistics

  • Wasserman, L. (2004). All of Statistics. Springer — probability, estimators, the bootstrap (Probability, Estimators, Uncertainty).
  • MacKay, D. (2003). Information Theory, Inference and Learning Algorithms. Cambridge University Press — full text free online (Uncertainty, L-LM).

Language models

  • Jurafsky, D.; Martin, J. H. Speech and Language Processing (3rd ed. draft) — $n$-grams, embeddings (L-LM, L-Embed).
  • Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS (L-Att, L-Transformer).

Fairness

  • Barocas, S.; Hardt, M.; Narayanan, A. Fairness and Machine Learning: Limitations and Opportunities. MIT Press — full text at fairmlbook.org (L-Fair).