The Normal Equations

Linear Algebra (Matrices) · Axiom Academy

When has no solution, the residual must be orthogonal to — and that single fact derives . 1. Closest Means the Residual Drops Straight Down Every reachable vector lives in the plane . Our target floats above that plane, so no lands on it. Slide a candidate foot around the plane and watch the leftover stretch and shrink — it is shortest exactly when it meets the plane at a right angle . That perpendicular foot is the least-squares solution , and the leftover at that spot is the residual . the residual stands perpendicular to the plane 2. Perpendicular to Every Column Is an Equation "Orthogonal to the plane" means orthogonal to each column of A (the columns span the plane). Dotting a column with the residual is one entry of — so all those dot products vanishing at once is exactly . Watch each column test against the residual, hit zero, and the zeros assemble into one matrix equation. The columns of A span , so the plane every column. is the i -th entry of — one column, one equation. Stacking every " =0 " gives the single vector equation . Distribute : , then send one term across. 3. Solve the Small System — and Land on the Foot The normal equations are only an system — tiny next to the original. Take the inconsistent system below: forming and gives a system we solve for . When is invertible, — and plugging it back, lands precisely on the projection foot from Step 1.

This is the written version of the interactive lesson above. See the full Linear Algebra (Matrices) course.