Overdetermined Systems

Linear Algebra for Machine Learning · Axiom Academy

More equations than unknowns: When exact solutions don't exist An overdetermined system has more equations than unknowns: m > n in the system Ax = b. In 2D with 3 lines, they typically don't all meet at one point: Real-world systems are almost always overdetermined: Data fitting: 1000 data points, fit a 5-parameter model → 1000 equations, 5 unknowns Least squares regression: Each measurement becomes one equation Machine learning: Training data defines constraints on model parameters Engineering: Multiple measurements, each with noise Since Ax = b has no exact solution, we find x that minimizes the residual error:

This is the written version of the interactive lesson above. See the full Linear Algebra for Machine Learning course.