Mod-ViT: A Vision Transformer-Based Framework for Breast Cancer Detection from Multiple Imaging Modalities

The evaluation of breast cancer plays a crucial role in its diagnosis through various imaging modalities. Although Convolutional Neural Network-based deep learning (DL) methods have significantly advanced this process, they are hindered by high computational demands and restricted spatial encoding capabilities. In contrast, Vision Transformer-based DL offers a compelling alternative due to its superior ability to encode spatial information efficiently. This study presents a breast cancer detection framework using a modified vision transformer (Mod- ViT) architecture across four publicly available datasets; mammogram (InBreast and MIAS) and ultrasound (BUSI and Mendeley). The workflow involves three key stages: data collection, image preprocessing, and classification. Mod- ViT utilizes transfer learning by modifying the pretrained vision transformer with additional layers, including flattening, batch normalization, dense, dropout, and a fully connected classification layer. The proposed Mod- ViT model was evaluated on each dataset, yielding superior performance metrics compared to VGG-16, ResNet-50, and VGG-19. On the InBreast dataset, Mod-ViT achieved 98.74% accuracy, 96.54% precision, 97.62% recall, and 98.25% F1-score, highlighting its ability to capture global mammogram features effectively. Similarly, on MIAS, BUSI, and Mendeley datasets, Mod- ViT consistently outperformed other models with accuracies of 99.92 %, 98.61 %, and 99.31 %, respectively. These results demonstrate Mod-ViT's robustness in classifying benign, malignant, and normal cases across diverse imaging modalities. By employing advanced attention mechanisms and effectively processing spatial and contextual information, Mod- ViT has shown its capability to address the complexities of breast cancer detection. This work highlights the promise of transformer-based models in enhancing diagnostic precision and reliability in medical imaging.