PCA Formulas & Reference

Linear Algebra for Machine Learning · Axiom Academy

PCA & Dimensionality Reduction Essential formulas for the unit Subtract the mean to center data at the origin. Divide by standard deviation for equal scale across features. Covariance Between Two Features d × d symmetric matrix of all pairwise variances and covariances. Principal Component Analysis (Linear) Eigenvectors are principal components; eigenvalues are variances. Express covariance matrix as product of eigenvectors and eigenvalues. Fraction of total variance captured by component i. Total variance retained using first k components. Data Projection onto Principal Components Transform centered data to k-dimensional PCA space. Approximate original data from k-dimensional representation (with loss). Factor data matrix directly without computing covariance matrix. Relationship Between SVD and PCA Singular values relate to eigenvalues; right singular vectors are PCs. Maps data implicitly to higher dimension. Gaussian kernel with bandwidth γ. Degree d polynomial relationships. n × n matrix of pairwise kernel evaluations; use for Kernel PCA instead of covariance matrix. Convert pairwise distances to conditional probabilities using Gaussian. Student-t distribution for heavy tails, preventing overcrowding. Objective to minimize; measures dissimilarity between two distributions.

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.