Conformalized classifiers with reject option

In data-driven decision support, predictive models built using machine learning aid in making informed decisions. In this context, models with a reject option may refrain from making predictions for certain instances. Accurately assessing the trade-off between predictive performance and throughput requires the ability to estimate performance at different rejection levels in advance. In this paper, we demonstrate how conformal prediction can be used for this purpose. Under exchangeability, the proposed conformalized classifiers can perfectly estimate accuracy or precision for any rejection level. In an empirical investigation using 41 publicly available datasets, the conformalized classifiers with a reject option are shown to clearly outperform probabilistic predictors calibrated with state-of-the-art techniques.