Understanding Singular Values

Linear Algebra for Machine Learning · Axiom Academy

Understanding the Importance and Behavior of Singular Values All singular values are always non-negative real numbers: By convention, singular values are arranged in decreasing order: where r ≤ min(m,n) is the rank of the matrix. Property 3: Connection to Matrix Rank The number of non-zero singular values equals the rank of the matrix: Example 1: Computing Singular Values Example 2: Singular Values and Matrix Norm The spectral norm (largest singular value) measures the maximum stretching a matrix applies: What Do Singular Values Tell Us? Data Perspective: In machine learning, if your data matrix A has singular values like [10, 5, 0.1, 0.05], it means the data primarily lives in 2-3 dimensions, with the rest being noise. This insight enables dimensionality reduction. Numerical Perspective: The ratio σₘₐₓ/σₘᵢₙ (the condition number) tells us how sensitive the system is to perturbations. Large ratios mean the system is ill-conditioned. Compression Perspective: By keeping only the top k singular values, we can compress data while preserving most of the important information in the top components.

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.