Data-driven random projection and screening for high-dimensional generalized linear models

We address the challenge of correlated predictors in high-dimensional generalized linear model (GLMs), where regression coefficients range from sparse to dense, by proposing a data-driven random projection (RP) method. This is particularly relevant for applications where the number of predictors is (much) larger than the number of observations and the underlying structure—whether sparse or dense—is unknown. We achieve this by using ridge-type estimates for variable screening and RP to incorporate information about the response–predictor relationship when performing dimensionality reduction. We demonstrate that a ridge estimator with a small penalty is effective for RP and screening, but the penalty value must be carefully selected. Unlike in linear regression, where penalties approaching zero work well, this approach leads to overfitting in non-Gaussian families. Instead, we recommend a data-driven method for penalty selection. In a simulation study, this data-driven RP improved prediction performance over conventional RPs, even surpassing benchmarks like elastic net. Furthermore, an ensemble of multiple such RPs combined with probabilistic variable screening delivered the best aggregated results in prediction and variable ranking across varying sparsity levels in our simulation study at a rather low computational cost. Final, three applications with count and binary responses demonstrate the method’s advantages in interpretability and prediction accuracy.