Multisource domain training with retinal expert-in-the-loop for accurate diabetic retinopathy staging from fundus images

Open-source datasets of digital fundus images annotated for diabetic retinopathy (DR) staging often suffer from inconsistent or inaccurate labeling but manually relabeling them at scale is impractical. A key need is a semi-supervised methodology which automatically flags images most likely to be mislabeled for expert review. We present DRStageNet2, our model improvement to DRStageNet, which results exclusively from our data refinement strategy with an expert-in-the-loop who reannotates images selectively in a data-driven manner across multiple rounds of model retraining and inference. Through this process, the expert reviewed a total of 3984 images among the 91 984 images used (2.8%). The expert modified the labels a total of 2592 (65%) of the reviewed images. DRStageNet2 improved out-of-domain generalization performance across five external validation sets. DRStageNet2’s multiclass accuracy of International Clinical Diabetic Retinopathy Scale scores 0–4 improved across all datasets with a mean increase of 3.6%. Its quadratic-weighted Cohen kappa score (Q-kappa) increased from an initial average of 0.89 (range: 0.86–0.91) to 0.91 (range: 0.89–0.93) with improvements of up to 6.2% in one dataset. DRStageNet2 then, achieves a Q-kappa score at the upper end of previously reported inter-rater agreement between retinal specialists.