Confidence estimation of AI model output for bladder cancer treatment response assessment in CT urography

Confidence estimation of machine learning (ML) and artificial intelligence (AI) model outputs is crucial for guiding decision-making. This study proposes two confidence estimation methods to enhance the trustworthiness of ML and AI models for classifying complete and non-complete responders for bladder cancer following neoadjuvant chemotherapy using CT urography: Segmentation Variability-Based Confidence Estimation (SVCE) and Classification Model Variability-Based Confidence Estimation (CMVCE). SVCE assesses prediction confidence based on the variability in predictions from models with different tumor segmentations, and thus extracted feature values, but the same classification model, while CMVCE evaluates confidence based on the variability in predictions from models with the same segmentations but different classification models. Our results demonstrate that cases with lower prediction variability consistently achieve better classification performance, with area under the receiver operating characteristic curve (AUC) values exceeding those of cases with higher variability. This study highlights that our proposed variabilitybased methods have the potential to provide an indicator of trustworthiness of the ML/AI prediction, which may enhance clinicians’ confidence in adopting ML/AI models for decision support.