PCA Derivation & Motivation

Linear Algebra for Machine Learning · Axiom Academy

PCA Derivation: Variance Maximization Finding principal components through optimization Objective: Maximize Variance of Projection The core idea of PCA is to find the direction w in which our data is most spread out (highest variance). Let's derive this mathematically. This centers our coordinate system at the data's center of mass. From now on, when we talk about variance, we're measuring spread around the origin. Step 2: Project Data onto Direction w This gives us a scalar value—the "coordinate" of the point along direction w . Step 3: Variance of Projection This measures how spread out the projected points are along direction w . Simplification Using Covariance Matrix We can simplify this variance expression. Since the data is centered: Where Σ is the covariance matrix we learned about earlier. This is a key insight: the variance of the projection depends on the direction and the covariance structure of the data. This constraint prevents trivial solutions. Step 5: The Optimization Problem In words: Find the unit vector that maximizes the variance of the data when projected onto it. Solution via Lagrange Multipliers We use the method of Lagrange multipliers. Define the Lagrangian: Where is the Lagrange multiplier. Step 7: The Eigenvalue Equation The covariance matrix has d eigenvectors (for d features). Which one maximizes variance? The variance along direction w equals the eigenvalue λ . Therefore:

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.