ITAG: An Image-Text Attention-Gated Network for Breast Tumor Classification Using Dynamic Optical Imaging

Breast cancer is one of the most prevalent and deadly cancers among women worldwide, with rising incidence and mortality rates, particularly in low-income regions. Conventional diagnostic methods such as mammography, ultrasound, and MRI have limitations, especially in early-stage cancer detection. This paper proposes an innovative approach using Dynamic Optical Breast Imaging (DOBI) combined with deep learning techniques for enhanced breast cancer diagnosis. We introduce the Image-Text Attention-Gated Classification Network (ITAG), which integrates multimodal data, including DOBI images and patient-specific textual information such as age and cup size. The ITAG model uses Vision Transformers (ViT) for image feature extraction and Transformer-based text encoders to process patient data. A multi-head attention-gated fusion network (MAGFN) optimizes the interaction between image and text features, improving the accuracy of tumor classification. Our experimental results demonstrate that ITAG outperforms existing methods, achieving an accuracy of 0.79, a sensitivity of 0.78, and an F1-score of 0.74, marking a significant advancement in the accuracy and robustness of breast cancer diagnosis.