Linear Regression

Linear Algebra (Matrices) · Axiom Academy

LESSON Linear Regression Is Least Squares Fitting the best line to data is exactly the normal-equations problem — same machinery, applied to points. 1. What "Best Fit" Means: Squared Residuals Pick any candidate line through the data. For each point (x_i, y_i) , the residual is the vertical gap to the line, . "Best fit" means making these gaps small — but we measure them squared , so each residual literally becomes the area of a square sitting on the point. Adding up those square areas gives the quantity we minimize — the sum of squared residuals (SSR): 2. Minimizing the Total Squared Area Now tilt and shift the line. As and change, every residual square changes size, and so does their total area . Watch the line sweep through candidates: the combined squared area shrinks to a single minimum — and the line sitting at that minimum is the least-squares fit. The squares on the far points blow up; SSR is large. The total squared area is as small as it can possibly get — no nearby line does better. Because is a sum of squares, it is a convex (bowl-shaped) function of . Setting both partial derivatives to zero pins the bottom of the bowl: 3. The Normal-Equations Machine Here is the payoff. Stack the data: a design matrix A whose first column is all ones and whose second column is the x_i , and an observation vector of the y_i . The best-fit coefficients solve . Watch and assemble from the data, then the solve drops out and the line snaps to its best fit.

This is the written version of the interactive lesson above. See the full Linear Algebra (Matrices) course.