Exact Leave-One-Out Cross-Validation for Ridge Regression at the Cost of One Fit: Proofs of the PRESS and GCV Formulas and a Test on Five Engineering Datasets

Leave-one-out cross-validation (LOOCV) normally needs n refits. For ridge regression, and any penalized least-squares fit with a fixed penalty, the leave-one-out residual equals eᵢ/(1−hᵢᵢ), where eᵢ is the ordinary residual and hᵢᵢ the leverage. I prove this with the Sherman–Morrison formula, including the case of an unpenalized intercept, show how one singular value decomposition gives LOOCV and generalized cross-validation (GCV) for every penalty at once, and test the result on five public UCI engineering datasets (concrete strength, airfoil noise, building heating load, yacht resistance and power-plant output) with full quadratic features. Against brute-force refitting, the shortcut matches to a relative error of at most 1.7×10⁻¹² on four datasets and 1.0×10⁻⁷ on the fifth, whose design matrix is numerically singular (condition number 1.1×10¹⁹). The whole 61-value penalty path took 1.3 to 10.9 ms, between 909 and 43,520 times faster than brute force. LOOCV and GCV chose the same penalty on the full data in all five cases. In 20 random 80/20 splits ridge improved test error only where the problem was small and ill-conditioned: yacht resistance, test RMSE 4.25 against 4.43 for the least-penalized fit. On the 9,568-point power-plant data the gain was 0.0001 MW.