Overfitting and Underfitting

Mathematical Modeling · Axiom Academy

LESSON Overfitting and Underfitting - Mathematical Modeling Unit 7: Data-Driven Modeling - Mathematical Modeling When fitting a model to data, we face a fundamental challenge: finding the right balance between a model that is too simple and one that is too complex. "A model should be as simple as possible, but no simpler." - Often attributed to Albert Einstein This balance is known as the bias-variance tradeoff , and understanding it is essential for building effective predictive models. Overfitting: When Models Memorize Overfitting occurs when a model learns not only the underlying pattern in training data but also the random noise. The model becomes too tailored to the specific training examples. Consider fitting a polynomial to n data points. A polynomial of degree n-1 can pass through all points exactly: Training error: 0 (perfect fit) But this polynomial will likely perform terribly on new data! Example: Polynomial Regression Suppose we have 10 data points generated from where is random noise. The degree-9 polynomial has near-zero training error but terrible test error - a classic sign of overfitting. Training error much lower than validation/test error Model parameters with very large magnitudes Predictions are unstable - small changes in data cause large changes in model Model complexity is high relative to amount of data Underfitting: When Models Oversimplify

This is the written version of the interactive lesson above. See the full Mathematical Modeling course.