Covariance Matrix

Linear Algebra for Machine Learning · Axiom Academy

Understanding relationships between features Step 1: Variance of a Single Feature Before we can understand how features relate to each other, let's recall how variance measures spread for a single feature. For a single variable x , the variance is: The variance tells us: on average, how much does a feature deviate from its mean? Step 2: Covariance Between Two Features Covariance measures how two features vary together. Do they tend to increase together, decrease together, or move independently? Step 3: Building the Covariance Matrix Σ For a dataset with d features, the covariance matrix Σ is a d × d matrix where: Step 4: Properties of the Covariance Matrix The covariance matrix is symmetric: This means if you swap rows and columns, the matrix stays the same. Property 2: Positive Semi-Definite (PSD) All eigenvalues of the covariance matrix are non-negative (≥ 0). This ensures that variance is never negative. Property 3: Real and Diagonalizable Because Σ is symmetric, it can always be decomposed into eigenvalues and eigenvectors (spectral theorem). This is crucial for PCA! Example: Computing a Covariance Matrix Consider a small dataset with 3 samples and 2 features (height in cm, weight in kg): Mean height = (170 + 180 + 160) / 3 = 170 cm Mean weight = (65 + 75 + 55) / 3 = 65 kg Var(Height) = (0² + 10² + (-10)²) / 3 = 66.67 Var(Weight) = (0² + 10² + (-10)²) / 3 = 66.67 Cov(Height, Weight) = (0·0 + 10·10 + (-10)·(-10)) / 3 = 66.67

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.