Radiomics Applicability Domain Analysis Classification Framework (RADAN-CF): A method for evaluating prediction reliability in radiomics

BACKGROUND AND OBJECTIVE: Radiomics-based machine learning models hold promise for clinical decision support, yet their deployment may be limited by the lack of transparent, prediction-level reliability assessment, especially under distributional shift. Existing uncertainty estimation methods mainly operate in probability space and may fail to identify unreliable predictions when test samples differ structurally or functionally from the training data. To address this gap, we propose the Radiomics Applicability Domain ANalysis - Classification Framework (RADANCF), a diagnostic approach for assessing the reliability of individual predictions in radiomics classification. METHODS: RADANCF integrates six binary reliability criteria spanning two domains: data representativeness (A-C), describing the relationship between test samples and the training data manifold, and model behavior (D-F), capturing local inconsistencies in predictive responses. Criteria violations are aggregated into ordered reliability categories summarized using a qualitative traffic-light scheme. The framework was evaluated on six public radiomics datasets using five machine learning classifiers, resulting in 900 model configurations trained under a dissimilarity-based stratified partitioning strategy designed to challenge model generalization. Analyses included prediction-level error modeling, multiway ANOVA, correlation analysis between criteria, and assessment of frequently violated criterion combinations. External validation was performed on an independent cohort of 2689 prostate cancer patients from the ProCAncer-I project. RESULTS: Prediction error was significantly associated with RADANCF category, although the relationship was not strictly monotonic, with intermediate categories showing the largest error contributions. RADANCF criteria were largely complementary, as shown by low pairwise Spearman correlations (only 7.5% of cases with correlations higher than 0.5; p < 0.001). Multiway ANOVA confirmed RADANCF category as a significant factor after controlling for dataset and model effects (p < 10⁻¹²). Specific combinations of broken criteria-particularly A, B, C, and E-were significantly overrepresented among higher-error predictions (Wilcoxon test, p < 0.001). In external validation, correct predictions appeared across all traffic-light categories, confirming the diagnostic and risk-oriented nature of RADANCF. CONCLUSIONS: RADANCF provides a transparent, per-prediction diagnostic framework for assessing reliability in radiomics classification under distributional shift. By jointly accounting for data representativeness and model behavior, it complements traditional performance and uncertainty metrics and supports more cautious model deployment in radiomics-based models.