An intelligent decision-making system integrating knowledge and data for image-based medical diagnosis

Deep learning (DL) methods utilize large numbers of images and annotating them is very labor-intensive. In contrast, in clinical practice such large numbers are not needed. Under the DL framework, how to integrate knowledge and data is unknown. We established an ensemble deep learning system (EDLS) that could integrate knowledge and data to automatically detect Glaucomatous Optic Neuropathy (GON), pathologic myopia and diabetic retinopathy using fundus images. An EDLS for the classification of GON was developed using 4225 fundus images obtained from Beijing Tongren Hospital. The generalization of the EDLS was tested on three testing datasets. Two EDLSs for the classification of pathologic myopia and diabetic retinopathy were developed and tested on two datasets obtained from websites respectively. For the classification of GON, the area under the receiver operating characteristic curve (AUC) of EDLS was 0.998 (95% CI, 0.997–0.999), with sensitivity of 97.5% and specificity of 98.3% on Testing dataset 1; the AUC of EDLS was 0.998 (95% CI, 0.997–0.999), with sensitivity of 93.7% and specificity of 99.8% on Testing dataset 2; and the AUC of EDLS was 0.984 (95% CI, 0.979–0.988), with sensitivity of 97.3% and specificity of 95.2% on Testing dataset 3. For the classification of pathologic myopia, the AUC of EDLS was 0.990 (95% CI, 0.982–0.999), with sensitivity of 91.5% and the specificity of 97.8%; for the classification of diabetic retinopathy, the AUC of EDLS was 0.916 (95% CI, 0.854–0.977), with sensitivity of 87.9% and specificity of 95.5%. Application of EDLS to fundus images from different settings demonstrated a high sensitivity, specificity, and generalizability for detecting GON, pathologic myopia and diabetic retinopathy.