A Comparative Study of Penalized and Classical Variable Selection Methods

In linear regression models, multicollinearity affects regression parameter estimates and can lead to misleading results in selecting the true model. This study, therefore, undertakes a comparison of the performance of various variable selection methods under multicollinearity. This simulation study evaluates the performance of classical criteria (Cp, AIC, AICC, ICOMP) and penalized regression techniques (LASSO, Ridge, Elastic Net, SCAD), as well as Partial Least Squares Regression (PLSR) for variable selection. The primary objective is to assess each method's effectiveness in accurately identifying the true model across varying levels of variance both in the presence and absence of multicollinearity. The results indicate that, in scenarios of multicollinearity, penalized methods and PLSR tend to yield results similar to those obtained in the absence of multicollinearity in the case of low variances, where they perform well in accurately estimating the true coefficients. In the presence of multicollinearity, Ridge regression emerges as the most successful method at a variance level of σ2=25. Moreover, ICOMP consistently outperforms classical selection criteria in settings of multicollinearity and high variance, while criteria such as Cp, AIC, and AICC exhibit diminished performance in this case.