BCS-Net: Multi-Task Breast Cancer Screening Network Enhanced by Multi-Modality Attention

This study aims to enhance the performance of computer-aided screening systems in early breast cancer screening by adopting multi-modality and multi-task strategies. Therefore, we propose BCS-Net: Multi-Task Breast Cancer Screening Network Enhanced by Multi-modality Attention, which targets three key tasks: segmentation of breast tumors, classification of mammography properties, and classification of tumors as benign or malignant. This study utilizes the Medio-Lateral Oblique (MLO) image and their corresponding diagnostic reports interpreted by our radiologists to conduct image-text alignment tasks using a simplified CLIP, thereby establishing an approach that integrates a multi-modality framework of images, text, and coordinates. The Text Guide Fusion Module (TGFM) enhances the performance of the backbone for downstream tasks on other unlabeled breast cancer datasets. The effectiveness of the proposed framework is verified through experiments, demonstrating not only an improvement in the accuracy and precision of breast cancer diagnosis, but also a comprehensive solution for computer-assisted early breast cancer screening. Our codes and breast cancer captions are available at https://github.com/None2211/multimodal_task.