A Probabilistic Framework for Class-specific Evaluation of Diagnostic Systems

Artificial intelligence (AI) systems are increasingly embedded in decision-making, yet classifiers are still commonly evaluated using aggregate metrics such as accuracy, F1-score or ROC AUC. These aggregate scores do not reveal whether predictions are made with adequate confidence, nor how uncertainty and errors are distributed across classes. We introduce a probabilistic evaluation framework that replaces the standard confusion matrix with a probabilistic confusion matrix and decomposes its probability mass into a confidence–correctness matrix with four interpretable components: reliability ( \(\mathrm{R}\) ), overconfidence ( \(\mathrm{O}\) ), underconfidence ( \(\mathrm{U}\) ) and ambiguity ( \(\mathrm{A}\) ). These quantities can be computed globally and per class, yielding class-specific profiles that distinguish decisive correct predictions, high-confidence errors, diffuse low-confidence behaviour and ambiguous near-miss cases. We evaluate the framework on 19 high-dimensional genomic cancer datasets from the CuMiDa repository. For the reference random forest classifier, conventional accuracy suggests strong performance on average ( \(\mathrm{Acc}=0.82\) ), whereas probabilistic accuracy is substantially lower ( \(\mathrm{Acc}^{\star}=0.64\) ), indicating weaker probability support for the true class. In the Leukaemia GSE28497 case study, the model attains \(\mathrm{Acc}=0.854\) , but only \(\mathrm{R}=0.545\) of the total probability mass corresponds to high-confidence correct predictions, with substantial underconfidence ( \(\mathrm{U}=0.360\) ). The proposed Class-specific Confidence–Correctness Diagram exposes tumour subtypes dominated by reliable predictions, high-confidence errors or diffuse uncertainty, providing a compact visual summary that complements reliability diagrams, ECE, NLL and Brier score. Additional experiments with logistic regression, multilayer perceptrons and post-hoc Platt scaling show that calibration and model choice can substantially redistribute probability mass even when hard-label performance changes little. Overall, the framework provides an interpretable, model-agnostic, probability-aware approach to evaluating diagnostic AI systems, aligning with emerging calls for uncertainty-aware, class-specific assessment in safety-critical applications.

Authors

Institutions

Publication Details

Journal
ACM Transactions on Knowledge Discovery from Data
Published
2026-11-21
DOI
https://doi.org/10.1145/3871044
Primary Topic
Imbalanced Data Classification Techniques
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

A Probabilistic Framework for Class-specific Evaluation of Diagnostic Systems

Jesús S. Aguilar–Ruiz
ACM Transactions on Knowledge Discovery from Data
Imbalanced Data Classification Techniques
article

A Probabilistic Framework for Class-specific Evaluation of Diagnostic Systems

Jesús S. Aguilar–Ruiz
article en

Abstract

Artificial intelligence (AI) systems are increasingly embedded in decision-making, yet classifiers are still commonly evaluated using aggregate metrics such as accuracy, F1-score or ROC AUC. These aggregate scores do not reveal whether predictions are made with adequate confidence, nor how uncertainty and errors are distributed across classes. We introduce a probabilistic evaluation framework that replaces the standard confusion matrix with a probabilistic confusion matrix and decomposes its probability mass into a confidence–correctness matrix with four interpretable components: reliability ( \(\mathrm{R}\) ), overconfidence ( \(\mathrm{O}\) ), underconfidence ( \(\mathrm{U}\) ) and ambiguity ( \(\mathrm{A}\) ). These quantities can be computed globally and per class, yielding class-specific profiles that distinguish decisive correct predictions, high-confidence errors, diffuse low-confidence behaviour and ambiguous near-miss cases. We evaluate the framework on 19 high-dimensional genomic cancer datasets from the CuMiDa repository. For the reference random forest classifier, conventional accuracy suggests strong performance on average ( \(\mathrm{Acc}=0.82\) ), whereas probabilistic accuracy is substantially lower ( \(\mathrm{Acc}^{\star}=0.64\) ), indicating weaker probability support for the true class. In the Leukaemia GSE28497 case study, the model attains \(\mathrm{Acc}=0.854\) , but only \(\mathrm{R}=0.545\) of the total probability mass corresponds to high-confidence correct predictions, with substantial underconfidence ( \(\mathrm{U}=0.360\) ). The proposed Class-specific Confidence–Correctness Diagram exposes tumour subtypes dominated by reliable predictions, high-confidence errors or diffuse uncertainty, providing a compact visual summary that complements reliability diagrams, ECE, NLL and Brier score. Additional experiments with logistic regression, multilayer perceptrons and post-hoc Platt scaling show that calibration and model choice can substantially redistribute probability mass even when hard-label performance changes little. Overall, the framework provides an interpretable, model-agnostic, probability-aware approach to evaluating diagnostic AI systems, aligning with emerging calls for uncertainty-aware, class-specific assessment in safety-critical applications.

ACM Transactions on Knowledge Discovery from Data
Universidad Pablo de Olavide (ES)
Imbalanced Data Classification Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon