Reliable Retinal Screening with Foundation Models under Distribution Shift

Abstract Reliable deployment of deep learning classifiers for retinal disease screening requires accurate in-distribution classification, effective out-of-distribution (OOD) detection, and well-calibrated predictions under distribution shift. This study investigates the open-world reliability of the EyeCLIP foundation model and evaluates whether calibration-oriented fine-tuning improves performance under external-domain shift. A three-class fundus classification task comprising diabetic retinopathy, glaucoma, and normal images was constructed using APTOS2019, G1020, and ORIGA, with ORIGA held out entirely for external evaluation. Standard balanced fine-tuning was compared with calibration-oriented fine-tuning and a frozen-feature linear probe. Five OOD scoring methods were evaluated alongside post-hoc temperature scaling and accept–reject analysis. Performance was assessed using classification, OOD detection, and calibration metrics. Standard balanced fine-tuning out-performed calibration-oriented fine-tuning in in-distribution accuracy (0.928 vs. 0.781), k-nearest-neighbor OOD detection (AUROC 0.938 vs. 0.878), and post-hoc OOD calibration (ECE 0.081 vs. 0.450). The calibration-oriented model misclassified 69% of normal ORIGA images as glaucoma. At 25% overall rejection, standard balanced fine-tuning rejected 50.9% of OOD samples while falsely rejecting 1.6% of in-distribution samples. Standard balanced fine-tuning combined with post-hoc calibration and feature-space k-nearest-neighbor abstention provides a simple and effective strategy for improving the reliability of open-world retinal screening.