Performance and Reliability Assessment of Deep Learning Models for OCT-Based Retinal Disease Classification

Abstract Background Deep learning models have achieved high classification accuracy on OCT images for discriminating among retinal diseases. However, clinical OCT datasets often exhibit substantial class imbalance, and aggregate metrics may obscure unreliable predictions for underrepresented classes. The reliability of such models for clinical decision support therefore remains uncertain. This study aimed to comprehensively assess the reliability of a deep learning-based OCT model under class imbalance. Methods Ten public OCT datasets were combined for a nine-class classification task of normal retina and eight retinal diseases. A Swin Transformer model was developed and evaluated on an internal test set, two independent public test sets (DUKE and OCTID), and a prospective clinical OCT cohort. Class-specific performance and bootstrap-derived 95% confidence intervals were analyzed alongside conventional metrics. Expected calibration error (ECE) assessed the reliability of predicted probabilities. LayerCAM was used to examine model attention to disease-related structures, and UMAP to visualized the deep feature space. Results On the internal test set, the model achieved a macro-AUC of 0.9992 and macro-recall of 0.9886. Despite high recall for some underrepresented classes, bootstrap estimates of precision and F1-score varied more widely, underscoring the need to consider sample size and uncertainty. Macro-F1 scores were 0.9884 on DUKE and 0.8944 on OCTID; recall for the underrepresented CSC class on OCTID was 0.7863. Pre-calibration ECE was 0.0048, indicating good agreement between predicted probabilities and observed outcomes. In the clinical cohort, macro-F1 was 0.8813 and macro-recall was 0.8365. LayerCAM activations corresponded spatially to disease-related retinal structures in many cases. UMAP showed class-specific clustering, with some overlap between classes. Conclusion This study assessed the reliability of deep learning-based OCT models for retinal disease discrimination under class imbalance. Overall performance alone is insufficient to characterize model reliability in multi-disease clinical settings. Class-specific stability, uncertainty, probability calibration, and interpretability also warrant assessment. These findings support a risk-aware decision-support strategy in which model outputs are interpreted together with class representation and uncertainty, and uncertain or underrepresented cases are flagged for specialist review. Prospective human–AI studies are needed before clinical deployment.