Choosing Number of Components

Linear Algebra for Machine Learning · Axiom Academy

Choosing the Number of Components How many principal components should you keep? Using more components preserves more information but defeats the purpose of dimensionality reduction. Using too few components loses important information. How do we decide? Step 1: Explained Variance Ratio For each principal component i, the explained variance ratio is: This tells us what fraction of total variance is captured by component i. Step 2: Cumulative Explained Variance The cumulative explained variance for the first k components: This tells us the total variance retained when using only the top k components. With 2 components, we explain 80% (not enough) With 3 components, we explain 95% (good!) With 4 components, we explain 100% (overkill) Another approach: plot the eigenvalues and look for an "elbow"—a point where the eigenvalues stop decreasing steeply. For the example above, the elbow is at k=3: PC1→PC2: drop from 50 to 30 (big decrease) PC2→PC3: drop from 30 to 15 (moderate decrease) PC3→PC4: drop from 15 to 5 (small decrease) ← elbow here So again, k = 3 is a good choice. Instead of manual selection, some libraries offer automatic methods: Once you've chosen k, you have a k-dimensional representation of your data. You can: Feed it to a downstream ML model Reconstruct the original data (with information loss)

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.