Classification Uncertainty Quantification: A Comparison Between Bootstrap and Conformal ROC Confidence Bands

Receiver Operating Characteristic (ROC) curves are commonly used to evaluate the performance of classification or prediction algorithms.However, in the literature, the uncertainty assessment of such algorithm is less rigorously addressed.In this article, we examine the limitations of the popular bootstrap method for ROC uncertainty quantification, elucidated by a simple yet essential model-based classification approach.We show that our proposed approach based on conformal prediction provides a valid solution for quantifying the uncertainty of the ROC curve and the Youden index.Both theoretical and numerical results corroborate the improved uncertainty quantification by the conformal inference over the bootstrap method.