Classification with reject option: Distribution-free error guarantees via conformal prediction

Machine learning (ML) models always make a prediction, even when they are likely to be wrong. This causes problems in practical applications, as we do not know if we should trust a prediction. ML with reject option addresses this issue by abstaining from making a prediction if it is likely to be incorrect. In this work, we formalise the approach to ML with reject option in binary classification, deriving theoretical guarantees on the resulting error rate. This is achieved through conformal prediction (CP), which produce prediction sets with distribution-free validity guarantees. In binary classification, CP can output prediction sets containing exactly one, two or no labels. By accepting only the singleton predictions, we turn CP into a binary classifier with reject option. Here, CP is formally put in the framework of predicting with reject option. We state and prove the resulting error rate, and give finite sample estimates. Numerical examples provide illustrations of derived error rate through several different conformal prediction settings, ranging from full conformal prediction to offline batch inductive conformal prediction. The former has a direct link to sharp validity guarantees, whereas the latter is more fuzzy in terms of validity guarantees but can be used in practice. Error-reject curves illustrate the trade-off between error rate and reject rate, and can serve to aid a user to set an acceptable error rate or reject rate in practice. • Proven Distribution-Free Error Rates for Singleton Predictions: Delivered theoretical guarantees on error rates for accepted singleton predictions, resolving inaccuracies found in prior works. • Refinement of Reject Option in Conformal Prediction: Clarified the relationship between error rate and reject rate using error-reject curves, offering a more accurate approach than existing models. • Practical Implementation Guidelines for Reject Option Models: Presented clear practical insights for implementing conformal predictors with reject options, providing empirical guidance for managing trade-offs in real-world applications. • Corrected Error Rate Analysis for Conformal Prediction with Reject Option: Provided a rigorous and corrected analysis of the error rate for binary classification with reject option, addressing weaknesses in previous studies.