Low-Rank Approximation

Linear Algebra for Machine Learning · Axiom Academy

Complete Worked Example of Rank-1 and Rank-2 Approximations Consider a 3 × 3 matrix A with SVD components: We want to find rank-1 and rank-2 approximations and compare them. Keep only the largest singular value: This uses only k=1 component. Notice how much simpler A₁ is compared to the full matrix A! Keep the two largest singular values: A₂ is more complex than A₁ but still much simpler than the full matrix A. Notice that the rank-2 error is smaller, as expected. By discarding only the smallest singular value, we retain more information. The total "energy" in matrix A is the sum of squared singular values: Key Insight: Trade-off Between Rank and Accuracy There's always a trade-off between: Storage/Computation: Lower rank k requires less storage and faster computation Accuracy: Higher rank k provides better approximation to the original matrix The goal is to find k such that: We compress sufficiently (small k) We retain enough information (error acceptable) Ideally, we want the discarded singular values σₖ₊₁, σₖ₊₂, ... to be very small In real data, singular values often decay rapidly. For example, in image data: σ₄ to σ₁₀₀ might be close to 0 In such cases, keeping just k=3 components gives excellent approximation while reducing computation from 100 dimensions to just 3!

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.