Principal Component Analysis Introduction

Linear Algebra for Machine Learning · Axiom Academy

PCA & Dimensionality Reduction Seeing the big picture in high dimensions The Challenge: High-Dimensional Data Imagine you have 1,000 features describing each of your data points. That's a 1,000-dimensional space—impossible to visualize, computationally expensive, and prone to overfitting. This is where Principal Component Analysis (PCA) comes in. PCA finds the directions in your data where the variance is largest—these are the directions where your data is most spread out, and they capture the most important structure. Intuition: Rotating to Find the Best View Think of your data as a cloud of points in space. If you rotate this cloud to align with the directions of maximum variance, you can project it onto a lower-dimensional space without losing much information. Watch as we rotate the data points. Notice how at certain angles, the data is stretched much more in one direction than the other. These are the principal components! Key Insight: PCA Finds Directions of Maximum Variance A high-resolution image has thousands of pixels. PCA can represent it with far fewer features while retaining visual quality. Each face image has thousands of pixel values, but faces lie on a low-dimensional manifold. PCA finds the principal "eigenfaces" that span this manifold. Scientists measure thousands of genes. PCA reveals which genes co-vary most, identifying the key patterns of genetic expression.

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.