Nonlinear Dimensionality Reduction

Linear Algebra for Machine Learning · Axiom Academy

Manifold learning and nonlinear dimensionality reduction As dimensionality increases, several bad things happen: Classification of Dimensionality Reduction Methods Principal Component Analysis (PCA) Finds linear directions of maximum variance. Fast, interpretable, but limited to linear structures. Linear Discriminant Analysis (LDA) Supervised version: finds linear directions that separate classes. Uses class labels. Uses kernel trick to find nonlinear components implicitly. Computationally expensive but effective. Preserve local or global structure. Includes t-SNE, UMAP, Isomap, Locally Linear Embedding (LLE). Consider a "Swiss roll" dataset (a 2D manifold embedded in 3D): Key Nonlinear Methods Overview Preserves geodesic distances (distances along the manifold). Works well when manifold is nearly convex. Pros: Preserves global structure, principled approach Cons: Slow, fails on highly nonconvex manifolds Locally Linear Embedding (LLE) Preserves local neighborhood structure. Each point is a linear combination of its neighbors. Pros: Fast, preserves local structure Cons: Doesn't preserve global structure, unstable on noisy data t-Distributed Stochastic Neighbor Embedding (t-SNE) Converts similarities to probabilities and minimizes divergence. Excellent for visualization. Pros: Beautiful visualizations, separates clusters Cons: Non-deterministic, slow, doesn't preserve global structure Uniform Manifold Approximation and Projection (UMAP)

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.