Condition Number & Numerical Stability

Linear Algebra for Machine Learning · Axiom Academy

Analyzing Numerical Stability and Sensitivity Using SVD The condition number κ(A) measures how sensitive the solution of Ax = b is to small perturbations in A or b. It tells us about the numerical stability of solving systems with matrix A. Computing Condition Number via SVD Using SVD, the condition number (spectral condition number) is: This is the ratio of the largest to smallest singular value. Example: Well-Conditioned vs Ill-Conditioned Well-Conditioned Matrix: Singular values close in magnitude Ill-Conditioned Matrix: Singular values vary dramatically The ill-conditioned matrix has κ = 100/1 = 100, while the well-conditioned has κ ≈ 1.22. This huge difference affects numerical stability! Condition Number Interpretation If we perturb A by ΔA or b by Δb, the relative error in the solution is approximately: Example: If κ = 100 and we perturb input by 0.01% (10 -4 ), the output error could be as large as 100 × 10 -4 = 0.01 (1%)! For ill-conditioned systems, even small data errors become large solution errors. Avoiding Numerical Problems with Ill-Conditioned Matrices Use SVD directly: SVD is the most numerically stable method for ill-conditioned systems Avoid normal equations: Computing A T A squares the condition number, making it much worse Regularization: Add a small regularization term to stabilize the solution Truncated SVD: Ignore small singular values to remove noise Double precision: Use higher precision arithmetic for very ill-conditioned systems

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.