Investigating the Generalizability of a Computer-Aided Detection System for Barrett’s Neoplasia by Application on Untrained Domains

Background and Objective: Neoplastic lesions in Barrett’s esophagus must be detected and resected endoscopically at an early stage due to their poor prognosis. In community hospitals, where many screening endoscopies are performed, a lack of expertise in detecting neoplastic lesions poses a problem. Computer-aided detection (CAD) systems can help address this issue. However, these algorithms are often trained using image data from a single center. In this thesis, we investigate the generalizability of such a system. We systematically analyse and evaluate its performance under varying conditions, including different endoscopy systems, endoscopy centers, and treatment status. Methods: An existing CAD system was applied to four test sets. Test Set 1 comprises images from the training center acquired using the same endoscopy system (160 images; 80 non-dysplastic Barrett’s esophagus (NDBE), 80 neoplastic). Test Set 2 contains images from the same center but acquired with a different endoscopy system (Olympus to Fujifilm; 524 images; 391 NDBE, 133 neoplastic). Test Set 3 also comprises images acquired with a new endoscopy system and, additionally, from a new center (213 images; 8 NDBE, 205 neoplastic). Test Set 4 consists of images of a previously treated esophagus (new treatment status); the center and endoscopy system are identical to Test Set 3. Performance is compared using statistical metrics. Results: A decline in performance was observed between Test Set 1 and 2 (sensitivity 91.25% vs. 37.59%; AUC 88.75% vs. 62.4%). Many neoplastic images are incorrectly classified as NDBE (FNR 62.41%). Between Test Set 2 and 3, sensitivity (37.59% vs. 68.78%) and AUC (62.4% vs. 71.89%, FNR 31.22%) increase. In Test Set 4, sensitivity and AUC are 64.86% and 65.41%, respectively, with an FPR of 34.03%. Conclusion: Due to small and unbalanced test sets, only limited conclusions can be drawn regarding the generalizability of the CAD system. Overfitting to the original training data, insufficient domain-specific training, and significant domain shifts could be reasons for the decline in performance. Before the CAD system can be used in new settings, image data from different endoscopy systems (e.g. Olympus), multiple medical centers and posttreatment esophagus should be integrated into the training. Alternatively, individualized models by fine-tuning with local hospital data could be developed. More diverse and larger training and test sets, standardized frameworks and more extensive external validation should be pursued to develop more robust and generalizable CAD systems.