Positive Definite Matrices

Linear Algebra for Machine Learning · Axiom Academy

The keys to optimization and stability Definition of Positive Definite A symmetric n×n matrix A is positive definite if: For all non-zero vectors x. This is saying: "applying the matrix to any vector always produces a positive result when you take the inner product." By the spectral theorem, checking if A is positive definite is easy: All eigenvalues must be strictly positive. No zero or negative eigenvalues allowed! Why this works: If A = QDQ^T with all positive eigenvalues on D, then: Since λ_i > 0 and y^T y > 0, the entire expression is positive. Positive semi-definite: All eigenvalues ≥ 0 (some can be zero) Negative definite: All eigenvalues < 0 Indefinite: Mix of positive and negative eigenvalues Why Positive Definiteness Matters If the Hessian (matrix of second derivatives) is positive definite, you have a strict minimum Gradient descent converges faster with positive definite Hessians Newton's method requires positive definiteness for stability Covariance matrices are always positive semi-definite Regularization terms (like L2 regularization) make loss functions more positive definite Kernel matrices in Support Vector Machines must be positive semi-definite Besides eigenvalues, there are other ways to check positive definiteness: Sylvester's criterion: All leading principal minors (determinants of upper-left sub-matrices) must be positive Cholesky decomposition: A can be written as A = LL^T where L is lower triangular with positive diagonal elements

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.