Machine Learning Lecture Notes

Simply put, machine learning is the part of artificial intelligence that actually works.

—Why Is Machine Learning (CS 229) The Most Popular Course At Stanford? Forbes

These webpages provide the lecture notes for the Machine Learning courses I teach at the University of Amsterdam and Amsterdam University College.

These notes are not meant to be a complete resource for learning about Machine Learning. These complement the lectures and other resources that are used in the courses. For all these resources I refer you to the Canvas webpages for your course.

Rein van den Boomgaard, January 2023

Contents:

• 1. Mathematical Tools
  • 1.1. Linear Algebra Recap
    • 1.1.1. Vectors
    • 1.1.2. Basis Vectors
    • 1.1.3. Linear Mappings
    • 1.1.4. Matrices
    • 1.1.5. Linear Equations, Determinants and Inverse Matrices
    • 1.1.6. Vector Norm, Inner Products and Orthogonality
    • 1.1.7. Eigenvectors and Eigenvalues
    • 1.1.8. Singular Value Decomposition
  • 1.2. Univariate Functions
    • 1.2.1. Definition
    • 1.2.2. Differentiating Univariate Functions
    • 1.2.3. Integrating Univariate Functions
    • 1.2.4. Scaling and Shifting Functions
    • 1.2.5. Extrema of Univariate Functions
    • 1.2.6. Exercises
  • 1.3. Multivariate Functions
    • 1.3.1. Plotting a multivariate function
    • 1.3.2. Differentiating a multivariate function
    • 1.3.3. Properties of Differentiation
    • 1.3.4. Integration of Multivariate Functions
    • 1.3.5. Symbolic Math Computations
    • 1.3.6. Exercises
  • 1.4. The Chain Rule of Differentiation
    • 1.4.1. A chain of functions
    • 1.4.2. The Chain Rule
    • 1.4.3. The Leibnitz Notation
    • 1.4.4. Multivariate Chain Rule
    • 1.4.5. Exercises
  • 1.5. Matrix Calculus
    • 1.5.1. The derivative of a scalar function with respect to a vector
    • 1.5.2. The derivative of a vector with respect to a scalar
    • 1.5.3. The derivative of a vector with respect to a vector
    • 1.5.4. Some important derivatives in the machine learning context
    • 1.5.5. Examples from Machine Learning
  • 1.6. Computational Graphs and the Chain Rule of Differentiation
    • 1.6.1. Computational Graph
    • 1.6.2. Automatic Differentiation
    • 1.6.3. Example
    • 1.6.4. Numerical Example
    • 1.6.5. Vectorial Graphs
    • 1.6.6. PyTorch
  • 1.7. Some Bits of Information Theory
    • 1.7.1. Entropy
    • 1.7.2. Bit Encoding
    • 1.7.3. Cross Entropy
    • 1.7.4. References
• 2. Probability and Statistics
  • 2.1. Probability Space and Probability Axioms
    • 2.1.1. Random Experiments and Probability Spaces
    • 2.1.2. Probability Axioms
    • 2.1.3. Venn Diagrams
    • 2.1.4. Simple Probability Theorems
  • 2.2. Conditional Probabilities
  • 2.3. Probability Trees
  • 2.4. Independent Events
  • 2.5. Random Variables
  • 2.6. Probability Distributions
    • 2.6.1. Discrete Distributions
    • 2.6.2. Continuous Distributions
  • 2.7. Joint Distributions
    • 2.7.1. Two Discrete Distributions
    • 2.7.2. Two Continuous Distributions
    • 2.7.3. Multiple Random Variables
  • 2.8. Calculations with Random Variables
    • 2.8.1. Scaled and Shifted Distributions
    • 2.8.2. The Sum of Random Variables
  • 2.9. Descriptive Statistics
    • 2.9.1. Histograms
    • 2.9.2. Statistics
  • 2.10. Estimators for Distribution Parameters
    • 2.10.1. Maximum Likelihood Estimator
    • 2.10.2. Bernoulli Distribution
    • 2.10.3. Categorical Distribution
    • 2.10.4. Normal Distribution
  • 2.11. Random Vectors
    • 2.11.1. Expectation and (Co)variance
    • 2.11.2. Normal Distributed Random Vectors
  • 2.12. Exercises
• 3. Overview of Machine Learning
• 4. Dimensionality Reduction
  • 4.1. Principal Component Analysis
    • 4.1.1. Maximizing Directional Variance
    • 4.1.2. A Linear Algebra View
    • 4.1.3. PCA in Practice
• 5. Regression
  • 5.1. Linear Regression
    • 5.1.1. Univariate Linear Regression
    • 5.1.2. Multivariate Linear Regression
    • 5.1.3. Dealing with Overfitting using Regularization
    • 5.1.4. Analytical Solution for Linear Regression
    • 5.1.5. Linear Regression from Basic Principles
    • 5.1.6. Exercises
• 6. Classification
  • 6.1. Bayesian Classification
    • 6.1.1. Maximum a Posteriori Classifier
    • 6.1.2. Naive Bayes Classifier
  • 6.2. Nearest Neighbor Classification
    • 6.2.1. Simple Nearest Neighbor Classifier
    • 6.2.2. k-Nearest Neighbor Classifier
    • 6.2.3. Weighted Nearest Neighbor Classification
  • 6.3. Logistic Regression
    • 6.3.1. Logistic Regression Model
    • 6.3.2. Maximum Likelihood Estimator
    • 6.3.3. Gradient Descent
    • 6.3.4. Vectorized Logistic Regression Algorithm
    • 6.3.5. Extended Features in Logistic Regression
    • 6.3.6. Multi Class Logistic Regression
  • 6.4. Neural Networks
    • 6.4.1. Neural Networks: the Third Revolution
    • 6.4.2. The Classical View on Neural Networks
    • 6.4.3. A Modern View On Neural Networks
• 7. Clustering
  • 7.1. k-Means Clustering
  • 7.2. k-Means, towards Gaussian Mixture Models
• 8. Methodology of Machine Learning
  • 8.1. Evaluating Classifiers