SAM-ColonPolypGen: Enhancing automated colon polyp report generation via reinforcement learning and prompt chaining

When polyps are detected during colonoscopy, endoscopists describe and record their characteristics based on endoscopic images, a process that is time-consuming and prone to error. Automated report generation has emerged as a promising approach to address this challenge. In this study, we propose SAM-ColonPolypGen, an automated model for colon polyp report generation that combines the Segment Anything Model (SAM) with a generative language model. Specifically, SAM-ColonPolypGen first extracts segmentation masks from endoscopic images using the SAM image encoder. These segmentation masks provide semantic information relevant to colon polyp description. Then the extracted masks are concatenated with textual prompts and provided as input to the generative language model to generate the corresponding colon polyp report. To further enhance the accuracy of our proposed model in describing colon polyp size and morphology, and to improve the professionalism of the generated reports, we integrate Reinforcement Learning (RL) and Prompt Chaining (PC) techniques. The entire framework was trained in an end-to-end manner. Comprehensive ablation studies and comparative experiments were conducted to evaluate the performance of the proposed SAM-ColonPolypGen model. Model assessment was performed using standard metrics, including METEOR, ROUGE, and BLEU, along with polyp size accuracy and polyp morphology accuracy. SAM-ColonPolypGen achieved the following results: METEOR 0.3961, ROUGE 0.6645, BLEU-1 0.5578, BLEU-2 0.4763, BLEU-3 0.4038, BLEU-4 0.3532, and polyp size accuracy of 62.00%, polyp morphology accuracy of 39.33%. These results indicate that SAM-ColonPolypGen outperforms baseline models, particularly in accurately describing polyp size, while also producing colon polyp reports that are more consistent and descriptive.