Least Squares Fitting

Mathematical Modeling · Axiom Academy

Learn how to find the best-fitting model to data by minimizing the sum of squared errors, the foundation of data-driven mathematical modeling. Suppose we have n data points and we want to fit a model function to this data. The residual at each point is the difference between the observed value and the model prediction: The least squares approach finds parameters that minimize the sum of squared residuals (SSR), also called the objective function: For a linear model with parameters m (slope) and b (intercept), we want to fit a line to data points. The objective function becomes: To minimize, we take partial derivatives with respect to m and b, set them equal to zero, and solve the resulting normal equations . For more complex models, we express least squares in matrix form. If we have a linear model with multiple parameters, we write the system as: Here, X is the design matrix containing the basis functions evaluated at each data point, and we seek the parameter vector that minimizes the residual norm. The least squares solution has a beautiful geometric meaning: it finds the point in the column space of X that is closest to y . The residual vector is orthogonal to the column space. The predicted values are a projection of y onto the column space of X: The residual is orthogonal to every column of X, which gives the normal equations: The total sum of squares decomposes into explained and residual parts:

This is the written version of the interactive lesson above. See the full Mathematical Modeling course.