Improved ridge estimation approach to address overfitting: Simulation and financial data analysis

Ridge regression is a popular biased estimation technique used to counteract multicollinearity, often preferred over ordinary least squares (OLS). A persistent challenge in ridge regression is the selection of an optimal penalty parameter to navigate the bias-variance trade-off, as no single ridge estimator performs uniformly well across diverse data conditions. To address this, we introduce three improved Ridge estimators (IREs) that dynamically calibrate the penalty parameter based on critical data characteristics: the intensity of multicollinearity and model dimensionality. Extensive Monte Carlo simulations, evaluated by mean squared error (MSE), demonstrate that IREs consistently surpasses existing methods, particularly in demanding scenarios marked by high multicollinearity, limited sample sizes, and elevated dimensionality. The practical utility and robustness of our approach are further confirmed through empirical applications to Longley data, establishing IRE as a valuable and reliable advancement in the penalized regression toolkit.