Bootstrapping the cross-validation estimate

Cross-validation is a widely used technique for evaluating the performance of prediction models, ranging from simple binary classification to complex precision medicine strategies. It helps correct for optimism bias in error estimates, which can be significant for models built using complex statistical learning algorithms. However, since the cross-validation estimate is a random value dependent on observed data, it is essential to accurately quantify the uncertainty associated with the estimate. This is especially important when comparing the performance of two models using cross-validation, as one must determine whether differences in estimated error are due to chance. Although various methods have been developed to make inferences on cross-validation estimates, they often have many limitations, such as requiring stringent model assumptions. This paper proposes a fast bootstrap method that quickly estimates the standard error of the cross-validation estimate and produces valid confidence intervals for a population parameter measuring average model performance. Our method overcomes the computational challenges inherent in bootstrapping a cross-validation estimate by estimating the variance component within a random-effects model. It is also as flexible as the cross-validation procedure itself. To showcase the effectiveness of our approach, we conducted comprehensive simulations and real-data analysis across two applications.