PCA via Eigendecomposition

Linear Algebra for Machine Learning · Axiom Academy

Using spectral decomposition to find principal components The Connection to Eigendecomposition From the previous module, we derived that finding principal components requires solving the eigenvalue problem of the covariance matrix: This is the starting point for PCA via eigendecomposition. The eigenvectors are our principal components, and the eigenvalues tell us the variance along each component. This means we can always decompose the covariance matrix as: Interpretation of Eigenvectors and Eigenvalues Eigenvectors = Principal Components Each eigenvector is a direction in the original feature space. It defines one principal component. Eigenvalues = Variance Explained Each eigenvalue tells us the variance of the data along the corresponding eigenvector. Larger eigenvalues mean more variance in that direction. By keeping only the eigenvectors with large eigenvalues, we preserve the most important structure in the data. The PCA Algorithm via Eigendecomposition Calculate the covariance matrix: Find eigenvectors and eigenvalues of . Let be the eigenvalues in descending order. Choose the top k eigenvectors. Store them as columns in a matrix (shape: d × k). The result is the data in the new k-dimensional PCA space. Consider a covariance matrix from some 2D data: Computing eigenvalues and eigenvectors of this matrix (using numpy or similar): First Principal Component (PC1): Direction: v₁ = [0.707, 0.707]ᵀ (diagonal direction, equally weighted combination of both features)

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.