Truncated SVD

Linear Algebra for Machine Learning · Axiom Academy

Low-Rank Approximation Using Truncated Singular Value Decomposition Instead of using all singular values in the decomposition A = UΣV T , truncated SVD keeps only the first k largest singular values and discards the rest. This creates a low-rank approximation of A. If A has SVD A = UΣV T , the rank-k truncated SVD approximation is: Uₖ is the first k columns of U (m × k matrix) Σₖ contains only the first k singular values (k × k diagonal matrix) Vₖ T contains only the first k rows of V T (k × n matrix) For an m × n matrix A with rank r, using k components (k ≤ r): Advantage: Original A requires mn storage. Truncated version requires k(m + n + 1) storage. For large m, n and small k, this is much more efficient. This is one of the most important results in linear algebra: Among all m × n matrices of rank at most k, the truncated SVD Aₖ = UₖΣₖVₖ T minimizes the Frobenius norm error ||A - Aₖ||_F. In plain English: Truncated SVD gives you the best possible low-rank approximation. If you want to approximate A using only k components, SVD is the optimal choice. No other method can do better. Reduce storage and transmission size while preserving important information. Remove noise and focus on the principal patterns in the data. Faster computation and reduced memory for subsequent operations. Project high-dimensional data to 2D or 3D for visualization. Choosing the right number of components k is crucial. Common strategies:

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.