A Probabilistic Framework for Class-specific Evaluation of Diagnostic Systems
Artificial intelligence (AI) systems are increasingly embedded in decision-making, yet classifiers are still commonly evaluated using aggregate metrics such as accuracy, F1-score or ROC AUC. These aggregate scores do not reveal whether predictions are made with adequate confidence, nor how uncertainty and errors are distributed across classes. We introduce a probabilistic evaluation framework that replaces the standard confusion matrix with a probabilistic confusion matrix and decomposes its probability mass into a confidence–correctness matrix with four interpretable components: reliability ( \(\mathrm{R}\) ), overconfidence ( \(\mathrm{O}\) ), underconfidence ( \(\mathrm{U}\) ) and ambiguity ( \(\mathrm{A}\) ). These quantities can be computed globally and per class, yielding class-specific profiles that distinguish decisive correct predictions, high-confidence errors, diffuse low-confidence behaviour and ambiguous near-miss cases. We evaluate the framework on 19 high-dimensional genomic cancer datasets from the CuMiDa repository. For the reference random forest classifier, conventional accuracy suggests strong performance on average ( \(\mathrm{Acc}=0.82\) ), whereas probabilistic accuracy is substantially lower ( \(\mathrm{Acc}^{\star}=0.64\) ), indicating weaker probability support for the true class. In the Leukaemia GSE28497 case study, the model attains \(\mathrm{Acc}=0.854\) , but only \(\mathrm{R}=0.545\) of the total probability mass corresponds to high-confidence correct predictions, with substantial underconfidence ( \(\mathrm{U}=0.360\) ). The proposed Class-specific Confidence–Correctness Diagram exposes tumour subtypes dominated by reliable predictions, high-confidence errors or diffuse uncertainty, providing a compact visual summary that complements reliability diagrams, ECE, NLL and Brier score. Additional experiments with logistic regression, multilayer perceptrons and post-hoc Platt scaling show that calibration and model choice can substantially redistribute probability mass even when hard-label performance changes little. Overall, the framework provides an interpretable, model-agnostic, probability-aware approach to evaluating diagnostic AI systems, aligning with emerging calls for uncertainty-aware, class-specific assessment in safety-critical applications.
Authors
- Jesús S. Aguilar–Ruiz (ORCID: https://orcid.org/0000-0002-2666-293X)
Institutions
- Universidad Pablo de Olavide (ES)
Publication Details
- Journal
- ACM Transactions on Knowledge Discovery from Data
- Published
- 2026-11-21
- DOI
- https://doi.org/10.1145/3871044
- Primary Topic
- Imbalanced Data Classification Techniques
- Type
- article
- Field-Weighted Citation Impact
- 0.00