Supervised-Residual Dimension Reduction for High-Dimensional Prediction

Abstract Supervised--Residual Dimension Reduction (SRDR) is proposed for high-dimensional regression and classification, where response-guided and variance-based predictor information may provide complementary predictive information. SRDR first constructs a response-guided block Krylov subspace and then estimates principal directions from the predictor variation remaining after projecting out the entire supervised subspace, rather than applying PCA to the original predictors. The resulting supervised and residual loading spaces are orthogonal, and cross-validation selects the Krylov order and residual dimension, allowing supervised-only, residual-only, or mixed representations. An equivalent formulation based on the $n\times n$ Gram matrix avoids forming a $p\times p$ covariance matrix. We establish exact finite-sample orthogonality, an exact primal--dual equivalence, and deterministic perturbation bounds for the resulting loading space. Simulation studies in regression and classification show that the selected allocation between supervised and residual directions depends on the underlying signal structure, with predictive gains when useful predictor variation remains outside the fitted supervised space. Applications with up to 32,321 predictors demonstrate computational feasibility for dense and sparse data and illustrate the selection of supervised-only and mixed representations.