Spectral Theorem

Linear Algebra for Machine Learning · Axiom Academy

The beauty of symmetric matrices The Spectral Theorem Statement Where Q is orthogonal (Q^T Q = I) and D contains the real eigenvalues on the diagonal. This is one of the most beautiful theorems in linear algebra! It guarantees that symmetric matrices are always diagonalizable, and moreover, their eigenvectors are orthogonal! Key Properties of Symmetric Matrices Real eigenvalues: All eigenvalues are real (no complex numbers) Orthogonal eigenvectors: Eigenvectors corresponding to different eigenvalues are perpendicular Complete orthonormal basis: We can form an orthonormal basis of eigenvectors Orthogonal diagonalization: A = QDQ^T (not Q^(-1), since Q^(-1) = Q^T for orthogonal matrices) The Spectrum and Eigenvalue Decomposition The "spectrum" of a matrix is the set of all its eigenvalues. For symmetric matrices, we can write: Where λ₁, λ₂, ..., λ_n are the eigenvalues and q₁, q₂, ..., q_n are the orthonormal eigenvectors. This is called the eigenvalue decomposition . Applications of the Spectral Theorem Covariance matrices in PCA: The covariance matrix is symmetric, so the spectral theorem guarantees orthogonal principal components Correlation matrices: Same as above, widely used in statistics Laplacian matrices in graph theory: Symmetric, used in spectral clustering Hessian matrices in optimization: The Hessian (second derivatives) is symmetric. Eigenvalues tell you about curvature! Quadratic forms: Analyze ellipsoids and understand data geometry

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.