A Novel Dataset for Polyp Segmentation and Detection Using a Vision-Language Model

Colorectal Cancer (CRC) is caused by malignant polyps that develop on the colon walls, and early detection is crucial for prevention. Colonoscopy is one of the most effective methods for the detection of polyps, and deep learning algorithms can be utilized for colonoscopy image processing. The main challenge with using deep learning models is the lack of suitable data to train the models. To address this, we introduce a colorectal polyp dataset with a larger statistical population than previous datasets. Moreover, we provide the first clinical reports of colonoscopy related to patients that can be used for the purpose of training visual-language models. To enhance segmentation accuracy, we propose a deep learning model that integrates a Global Attention Module (GAM) for improved feature extraction and a Multi-Scale Aggregation Module (MSAM) for efficient fusion of information. The GAM is designed to extract features from different scales while utilizing low-level features to enhance high-level feature extraction. Furthermore, we employ a vision-language model to analyze the impact of different textual information on performance, which is crucial for report generation tasks and multimodal learning. Results indicate that the proposed model performs better than state-of-the-art studies by 2.7% and 2.5%, respectively, in mDice and mIoU. The dataset and code are available at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/javadmozaffari/PolypDataset</uri>.