Analysis of Loss Functions for Colorectal Polyp Segmentation Under Class Imbalance

Class imbalance is a persistent limitation in polyp segmentation, commonly resulting in biased predictions and reduced accuracy in identifying clinically relevant structures. This study systematically evaluated 12 loss functions, including standard, weighted, and compound formulas, applied to colon polyp segmentation using the UNet-VGG16 fixed architecture on the Kvasir-SEG dataset. The encoder was frozen to isolate the effect of loss functions under the same training conditions. A fixed random seed was used in all experiments to ensure reproducibility and control variance during training. The results reveal that the combined loss functions, namely WBCE combined with Dice and Tversky combined with Focal, achieved the top Dice scores of 0.8916 and 0.8917, respectively. Tversky plus Focal also provided the highest sensitivity of 0.8885, and WBCE obtained the best average IoU of 0.8120. Tversky loss showed the lowest error rate of 4.99, indicating stable optimization. These results clarify the influence of loss function selection on segmentation performance in scenarios characterized by considerable class imbalance.