SVD and Pseudoinverse

Linear Algebra for Machine Learning · Axiom Academy

Computing the Moore-Penrose Pseudoinverse Using SVD For a non-square or singular matrix A that doesn't have an inverse, the pseudoinverse A + (also called the Moore-Penrose inverse) is a generalization. It satisfies: Computing Pseudoinverse via SVD The SVD provides the simplest way to compute A + : Taking the reciprocal of non-zero singular values: 1/σ₁, 1/σ₂, ..., 1/σᵣ Placing zeros where A had zero singular values Transposing the result if necessary Given a 2×3 matrix A (more columns than rows): Verification: Pseudoinverse Properties The pseudoinverse satisfies these properties: Application: Least-Squares Problems The pseudoinverse is crucial for solving least-squares problems. If the system Ax = b is overdetermined (more equations than unknowns), we want to find x that minimizes ||Ax - b||. Solution: x * = A + b gives the least-squares solution. This is the most numerically stable way to solve least-squares problems because SVD is backward-stable (unlike normal equations which can amplify numerical errors). Application: Rank-Deficient Systems When A is rank-deficient (singular), the ordinary inverse doesn't exist. But the pseudoinverse always exists! Example: A matrix with a zero singular value: σ₁ = 5, σ₂ = 2, σ₃ = 0 The pseudoinverse simply ignores the zero singular value and inverts the others. Numerically Stable: SVD is backward-stable and doesn't suffer from ill-conditioning Handles Rank Deficiency: Automatically deals with zero singular values

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.