Frobenius Norm & Eckart-Young Theorem

Linear Algebra for Machine Learning · Axiom Academy

Computing and Interpreting Matrix Norms Using Singular Values The Frobenius norm is a measure of matrix size. It's the square root of the sum of all squared entries: It's like taking the Euclidean norm if you treat the matrix as a vector (stack all entries). The Power of SVD: Computing Frobenius Norm Here's the beautiful part: using SVD, computing the Frobenius norm becomes trivial! Why this works: The Frobenius norm equals the Euclidean norm of the vector of singular values. We just need to sum the squares of the singular values! Method 2: Via SVD singular values σ₁=5, σ₂=3 Both methods give the same answer, but the SVD method is more efficient and numerically stable! When we truncate SVD to keep only k components, the Frobenius norm of the error is: Example: If σ₁=5, σ₂=3, σ₃=1, then: The error in the rank-2 approximation is just 1, which is very small compared to the original norm of √35 ≈ 5.9. To measure approximation quality, we often use relative error: A relative error of 3.8% means the rank-2 approximation captures 96.2% of the information. This is excellent compression! Frobenius norm: ||A||_F = √(σ₁² + σ₂² + ... + σᵣ²) Spectral norm (largest singular value): ||A||₂ = σ₁ Nuclear norm (sum of singular values): ||A||* = σ₁ + σ₂ + ... + σᵣ Rank: rank(A) = number of non-zero singular values Trace: tr(A T A) = σ₁² + σ₂² + ... + σᵣ² The Frobenius norm is crucial for understanding: Approximation quality: How well does our low-rank approximation preserve the original data?

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.