Standardization in PCA

Linear Algebra for Machine Learning · Axiom Academy

Why centering and scaling matter for principal component analysis Before computing PCA, your data must be properly preprocessed. The two key steps are: This seems simple, but the choice has profound implications for results! PCA maximizes variance around the origin. If data isn't centered, the first principal component will point toward the center of mass rather than the direction of maximum spread. This ensures the covariance matrix is computed correctly, with variances measured around the true center of the data. After centering, you can optionally standardize (divide by standard deviation): This decision fundamentally changes PCA results. Let's see why with an example. Worked Example: Height and Weight Height: Mean = 170 cm, Std Dev = 10 cm, Range = [160, 180] Weight: Mean = 66.67 kg, Std Dev = 7.5 kg, Range = [60, 75] Approach 1: Centered but NOT Standardized Variance of height: much larger (100 scale) Variance of weight: smaller (50-60 scale) PC1 aligns almost entirely with height Height dominates the first PC (80%+ variance) Weight gets minimal contribution Not meaningful: we're just following the larger-scale variable Approach 2: Centered AND Standardized Step 1: Center and Standardize Now height and weight have similar variance (both standardized to variance ≈ 1) PC1 is a meaningful combination of both features Roughly equal weight on height and weight PC1 captures overall body size (tall and heavy together)

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.