Least Squares Method

Linear Algebra for Machine Learning · Axiom Academy

Finding the best fit when exact solutions don't exist Given an overdetermined system (with no exact solution), find that minimizes: This is the sum of squared residuals (errors). Geometrically, least squares projects onto the column space of : From the perpendicularity condition, we derive the normal equations: Ubiquitous in science: Every experiment has measurement noise Statistically optimal: Under Gaussian noise, least squares is the maximum likelihood estimator Computationally efficient: Can solve in O(mn²) time (better than direct methods) Foundation of machine learning: Linear regression, neural networks, and more all use least squares concepts Interpretable: The solution minimizes a clear, understandable objective

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.