Mitigating Overfitting in Deep Learning: Insights from Bayesian Regularization

Overfitting is one of the most commonly faced challenges when designing a model, it's when model performs well on training data but poorly on unseen data. This review explores Bayesian perspectives on regularization techniques for machine learning models. Bayesian regularization incorporates prior knowledge into the fitting process, allowing for a more controlled and probabilistic understanding of the model parameters. An exploration into the theoretical foundations of Bayesian regularization is done by examining parametric and non-parametric Bayesian methods, analyze their potential to improve model generalization and reduce overfitting. Theoretical demonstration and well-known results on simple models and feed forward neural network are presented to emphasize the impact regularization can have on general accuracy and model's generalization capabilities. By synthesizing recent advancements, a set of insights regarding the integration of Bayesian frameworks with traditional regularization strategies is provided. This paper aims to guide future research in developing robust, Bayesian-informed regularization strategies.