PCA via SVD

Linear Algebra for Machine Learning · Axiom Academy

PCA via Singular Value Decomposition Computing principal components directly from the data matrix While eigendecomposition of the covariance matrix is elegant, it's computationally expensive for high-dimensional data. Singular Value Decomposition (SVD) is numerically more stable and faster. For a centered n × d data matrix , SVD decomposes it as: The right singular vectors (columns of ) are exactly the principal components! They form an orthonormal basis. And the singular values relate to the covariance eigenvalues by: This means the singular values squared (divided by n-1) give us the variance explained by each principal component. Subtract the mean from each feature: The principal components are the columns of . Keep the top k columns. 3 samples, 2 features (centered): Variance ratio from singular values: So PC1 explains 91.3% and PC2 explains 8.7% of the variance. SVD vs Eigendecomposition: When to Use Which

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.