Hybrid Loss-Driven Attention Residual U-Net with Multiscale Cascaded CBAM for Breast Cancer Segmentation

In medical imaging, breast tumor segmentation is the most difficult task for early detection. However, the task of segmenting tumors remains challenging because of low contrast, irregular shapes, fuzzy borders, and the presence of noise. To address these issues, a U-shaped encoder-decoder network is proposed that consists of the residual block with a Multi-Cascade Convolutional Block Attention Module (MC-CBAM) to segment the breast lesions. The MC-CBAM module contains channel attention, which focuses on the refinement of the most important channels, while spatial attention is responsible for the refinement of the spatial features by cascade convolution at various scales, including 1 × 1, 3 × 3, and 5 × 5. The architecture contains 4 encoder blocks and 4 decoder blocks, integrated with skip connections and having a bottleneck in between. The model is trained using chosen hyperparameters of 8 batch size, 50 epochs, and a learning rate of 1e-4. Additionally, a combined loss function is computed, which combines the Tversky loss and a Dice-based loss to improve the process of segmentation, especially for the small tumor regions. The effectiveness of the proposed framework is assessed on ultrasound and histopathology datasets. The outcomes demonstrate that the integration of the MC-CBAM block contributes to a better segmentation process. On the histopathology dataset called Triple Negative Breast Cancer tissues (TNBC), the model gained a Dice and Jaccard of 80.54% and 67.43%. While using the ultrasound datasets, the model provides the dice and Jaccard of 87.23% and 77.36% on the UDIAT dataset and 83.96% and 73.25% on the BUSI dataset, respectively. The results integrate the robustness across the different imaging modalities.