Cross-Validation

Mathematical Modeling · Axiom Academy

LESSON Cross-Validation - Mathematical Modeling Unit 7: Data-Driven Modeling - Mathematical Modeling Cross-validation is a resampling technique used to evaluate how well a model generalizes to independent data sets. It answers the critical question: "How will my model perform on data it has never seen?" Cross-validation is essential because: Training error alone is a poor estimate of true model performance It helps detect and prevent overfitting It enables fair comparison between different models It provides estimates of prediction uncertainty A model that performs perfectly on training data may fail on new data. Cross-validation helps us measure true predictive ability. Training Error vs Generalization Error The training error measures performance on the data used to fit the model. The generalization error measures expected performance on new, unseen data. Key insight: Training error typically underestimates generalization error, especially for complex models. As model complexity increases: Training error continues to decrease Generalization error first decreases, then increases The gap between them grows (the "generalization gap") Cross-validation estimates the generalization error without requiring a separate test set. The simplest validation approach: split data into training and test sets. Randomly split data into training set (e.g., 70-80%) and test set (20-30%) Fit the model using only the training set Evaluate model performance on the test set

This is the written version of the interactive lesson above. See the full Mathematical Modeling course.