Confidence Limits for Prediction Performance

As machine learning algorithms become increasingly integrated into critical systems, assessing the reliability of their predictions is essential, especially when errors can have severe consequences. Incorporating statistical methods can help to quantify the inherent uncertainty and improve decision-making. This work proposes a new method for estimating confidence limits for prediction performance. In the first part, we introduce fundamental concepts and findings from the machine learning and statistical inference literature, framing the selection and evaluation of prediction models as a statistical inference problem. In particular, we consider the simultaneous evaluation of multiple candidate models and interpret this as a multiple testing problem. We also explore the bootstrap and nonparametric bootstrap tilting, which provides a reliable approach for estimating confidence intervals without the need to assume a specific underlying distribution. The second part integrates these concepts and presents the proposed multiplicity- adjusted bootstrap tilting lower confidence limits for conditional prediction performance. This approach is computationally undemanding and universally applicable to any combination of prediction models, model selection strategies, and performance measures. We prove that the proposed interval asymptotically achieves the nominal coverage probability and conduct simulation experiments to assess its goodness in finite samples. Specifically, we investigate the prediction accuracy of lasso and random forest classifiers. The proposed approach shows reliable coverage and competitive lower confidence limits. In contrast, we also show that recent alternative methods such as bootstrap bias-corrected cross-validation and nested cross-validation may fail to accurately track conditional performance. Finally, we apply the proposed approach to real-world data, where it demonstrates stability when model selection is highly sensitive to the allocation of the sample data to the learning and evaluation sets or in the presence of a distribution shift.