Overfitting
A phenomenon where a model learns training data too well, including noise and idiosyncrasies, resulting in degraded performance on unseen data (poor generalization).
Overfitting occurs when a model memorizes training data noise rather than learning generalizable features, degrading prediction on new data. A classic symptom: training accuracy exceeds 99% while validation accuracy stagnates around 70%. It is more likely when model complexity is disproportionate to available data.
In image recognition, training a CNN with millions of parameters on a few thousand images can lead to overfitting.
- Data augmentation: Expands training data through rotations, flips, color jittering, and random crops
- Dropout: Randomly deactivates neurons during training (for example, 50%), preventing excessive co-adaptation of features. At inference, all neurons are active with scaled outputs
- Early stopping: Halts training when validation loss stops improving
- Regularization: L2 weight decay penalizes large weights, constraining complexity and encouraging simpler representations
Transfer learning is another countermeasure: a model pre-trained on large-scale data is fine-tuned on limited task-specific data.