Kernel PCA

Linear Algebra for Machine Learning · Axiom Academy

Nonlinear dimensionality reduction Imagine a 3D dataset that looks like a rolled-up piece of paper. The true underlying structure is 2D (the paper), but it's nonlinearly embedded in 3D space. Linear PCA would find 3 components instead of recognizing the 2D structure. Instead of working in the original feature space, we compute PCA in a higher-dimensional space implicitly, using the kernel trick. This allows us to find nonlinear principal components without explicitly computing coordinates in the high-dimensional space. 1. Radial Basis Function (RBF) Kernel γ is a bandwidth parameter. Larger γ means tighter clusters. Degree d determines the complexity. d = 2 or 3 is common. Similar to neural network activation. Compute , an n × n matrix where Adjust K to account for centering in the high-dimensional space (not trivial!) Find the leading eigenvectors of K (the Gram matrix) Map each point to the nonlinear principal components Slower O(n³) (depends on n, not d) There's no universal best kernel—it depends on your data. Some guidelines:

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.