Discussion 12: Principal Components Analysis (PCA)
Rayleigh quotients and their connection to the spectral norm and related optimization problems. Derivations of PCA through various methods: Gaussian MLE, maximizing variance, and minimizing projection error. Relationship between the SVD and PCA.
> Slides
Discussion 11: Neural Networks
Neural network basics: feature/representation learning, universal function approximation, motivations for backprop, and how to derive gradients for functions involving matrices and batch dimensions.
> Slides
Discussion 10: Kernel Methods
Kernel methods and their motivation as both enabling efficient high-dimensional featurization, and allowing custom notions of similarity between data points. Conditions for the validity of a kernel function.
> Slides
Discussion 9: Decision Trees & Random Forests
Decision tree foundations: entropy, information gain, and strictly concave cost functions. Motivation behind random forests.
> Slides
Discussion 7: Midterm Review
Miscellaneous practice problems: logistic regression, squared vs. logistic vs. hinge loss functions, LDA/QDA, gradient descent and convexity
Discussion 6: Least Squares & Least Norm
Least-squares linear regression and motivation for the min-norm solution in the case of infinitely many solutions. SVD, the Moore-Penrose Pseudoinverse, and its application to the min-norm least squares problem.
> Slides
Discussion 5: Anisotropic Gaussians, Transformations, Quadratic Forms
Overview of anisotropic Gaussians, including properties of the covariance matrix and the elliptical isocontours of the quadratic form. Change of basis as a way to understand various data transformations (sphering, whitening, etc.).
> Slides
Discussion 4: Generative Models, GDA, MLE
Review of Bayes Decision Theory and MLE, and their applications to generative modeling. Gaussian Discriminant Analysis (QDA/LDA) as a special case of generative models.
> Slides
Discussion 3: Soft-Margin SVMs, Decision Theory
Soft-margin SVMs, hinge loss, and interesting variants of SVMs for outlier detection. Deriving posterior class probabilities using Bayes' Rule.
> Slides
Discussion 2: Math Prep
Review of math concepts that are useful in machine learning: linear algebra, probability, and vector calculus (especially taking derivatives of matrix/vector functions).
> Slides
Discussion 1: Intro & SVMs [recording]
Review of vectors, projection, hyperplanes, and the distance formula. Intro to hard-margin SVMs, including motivation and formulation of the optimization problem.
Additional resources
Understanding the SVM formulation
