Bayesian Prediction of Future Empirical ROC Curves and AUC

Abstract The measured performance of a classifier can change when it is evaluated on a new test sample, even when the classifier itself remains fixed. We propose DiP-ROC (Dirichlet predictive ROC), a Bayesian framework for predicting this variation from a single test sample. It predicts the receiver operating characteristic (ROC) curve and its area under the curve (AUC) that a new sample from the same populations could produce, with the same numbers of positive and negative observations. DiP-ROC groups prediction scores into bins defined by a fixed threshold grid and uses a Dirichlet--multinomial model for each class. Together, these models describe the joint uncertainty in the future ROC points and their corresponding grid-based AUC. They preserve the dependence between ROC points, which are calculated from the same observations. They also account for both uncertainty about the score distributions and the variation caused by drawing a new test sample. The resulting summaries are simultaneous predictive bands for all ROC points on the grid and predictive intervals for the AUC. Prior information can come from an earlier evaluation of the same classifier. This information may be a reported AUC value or two score histograms, one for each class. Experiments on synthetic data and nine real-world datasets show how prior strength and class counts affect the reliability and precision of predictions for a future evaluation. All code is available on GitHub to support reproducibility: \url{https://github.com/ocaelen/bayesian_roc}.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-17
DOI
https://doi.org/10.21203/rs.3.rs-11270016/v1
Primary Topic
Imbalanced Data Classification Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Bayesian Prediction of Future Empirical ROC Curves and AUC

Olivier Caelen, Gabriele Iannucci
Research Square
Imbalanced Data Classification Techniques
preprint

Bayesian Prediction of Future Empirical ROC Curves and AUC

Olivier Caelen, Gabriele Iannucci
preprint en

Abstract

Abstract The measured performance of a classifier can change when it is evaluated on a new test sample, even when the classifier itself remains fixed. We propose DiP-ROC (Dirichlet predictive ROC), a Bayesian framework for predicting this variation from a single test sample. It predicts the receiver operating characteristic (ROC) curve and its area under the curve (AUC) that a new sample from the same populations could produce, with the same numbers of positive and negative observations. DiP-ROC groups prediction scores into bins defined by a fixed threshold grid and uses a Dirichlet--multinomial model for each class. Together, these models describe the joint uncertainty in the future ROC points and their corresponding grid-based AUC. They preserve the dependence between ROC points, which are calculated from the same observations. They also account for both uncertainty about the score distributions and the variation caused by drawing a new test sample. The resulting summaries are simultaneous predictive bands for all ROC points on the grid and predictive intervals for the AUC. Prior information can come from an earlier evaluation of the same classifier. This information may be a reported AUC value or two score histograms, one for each class. Experiments on synthetic data and nine real-world datasets show how prior strength and class counts affect the reliability and precision of predictions for a future evaluation. All code is available on GitHub to support reproducibility: \url{https://github.com/ocaelen/bayesian_roc}.

Research Square
Sapienza University of Rome (IT), UCLouvain (BE)
Imbalanced Data Classification Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon