GILED: Lesion Detection of Gastrointestinal Tract from Endoscopic Images and Medical Notes

Existing methods for lesion detection in medical images typically use images as the primary training source for deep models. However, in addition to annotated images, the doctor provides his observations and conclusions about the lesion in a medical note, which remains underutilized. This paper presents a method for automatic detection of gastrointestinal tract lesions, named GILED, which combines endoscopic images and medical notes. Specifically, we leverage the medical notes accompanying each medical image of a patient to enhance the representation of the image encoder. Once pre-trained on an image-text dataset, this enhanced image encoder can be integrated into any model for detection and segmentation. In our work, we deploy the image encoder in the YOLOv8 model for the lesion detection task. To demonstrate the effectiveness of this image representation learning approach, we conduct experiments on a benchmark dataset Kvasir-SEG and two self-collected datasets. Thank to the integration of expert knowledge from medical notes, GILED achieves higher performance even with limited image-text data sources. On the Kvasir-SEG dataset, compared to the original Yolov8, GILED improved the mAP50 from 91.5% to 92.8%; on our datasets, the mAP50 increased from 83.6% to 85.1% and 53.0% to 54.0%. It shows the potential to improve lesion detection using additional medical notes.