Balancing Act: Understanding Model Capacity in Neural Networks - From Overparameterization to Underfitting

Thrusting deep learning into the spotlight is a critical juncture at which determining model capacity and optimising its capacity are crucial not just to understanding deep learning, but to its pace of advancement. This comprehensive review examines two fundamental challenges: Between these two extremes, opposite ends of the model complexity spectrum, are overparameterization and insufficient model capacity. With neural networks growing in size and complexity to the point that some models contain billions of parameters, there has never been a more critical balancing act to strike between model capacity and performance. In this paper, I analyse systematically these challenges, their impact on modern machine learning systems, and current solutions and best practises for dealing with them. We draw from seminal works in the field, exploring the common ground in the impact of these seemingly contradictory problems on model performance and generalisation capabilities: having too few or too many parameters.