Residual Modelling as a New Approach for Variable Selection

Variable selection in statistical model building still has challenges to overcome as the depth and breadth of the research data is expanding. To help reduce this challenge, we introduce a new approach in variable selection, called residual modeling, which can be applicable regardless of the number of predictors. We compare the statistical power and type-1 error retainment of the forward, backward, and stepwise variable selection approaches with the proposed modeling strategy controlling for known predictors. In Residual Modeling, each predictor enters the model as a single predictor, whose resulting residuals become the dependent variable for the next predictor, and so on. We compare these models under different scenarios with varying sample sizes and various combinations of significant and insignificant predictors. When there exist known predictors from the literature, in identifying new significant predictors controlling for these known predictors, Residual Modelling shows higher statistical power especially as the number of predictors increases compared to the other variable selection methods used. It also has reduced bias in parameter estimation and reduced standard errors. The Type-1 error was retained at its nominal level for Residual Modelling while forward, backward, and stepwise variable selection approaches had slightly reduced Type-1 Error rates. When dealing with multiple predictors in the presence of known significant predictors, Residual Modelling offers a practical solution without causing loss of statistical power or increased Type-1 Error Rate.