A multi-task cross-attention strategy to segment and classify polyps

Polyps are the main biomarkers for diagnosing colorectal cancer. Their early detection and accurate characterization during colonoscopy procedures rely on expert observations. Nevertheless, such a task is prone to errors, particularly in morphological characterization. This work proposes a multi-task representation capable of segmenting polyps and stratifying their malignancy from individual colonoscopy frames. The approach employs a deep representation based on multi-head cross-attention, refined with morphological characterization learned from independent maps according to the degree of polyp malignancy. The proposed method was validated on the BKAI-IGH dataset, comprising 1200 samples (1000 white-light imaging and 200 NICE samples) with fine-grained segmentation masks. The results show an average IoU of 83.5% and a recall of 94%. Additionally, external dataset validation demonstrated the model's generalization capability. Inspired by conventional expert characterization, the proposed method integrates textural and morphological observations, allowing both tasks, polyp segmentation and the corresponding malignancy stratification. The proposed strategy achieves the state-of-the-art performance in public datasets, showing promising results and demonstrating its ability to generate a polyp representation suitable for multiple tasks.