Ensemble Architecture of Vision Transformer and <scp>CNNs</scp> for Breast Cancer Tumor Detection From Mammograms

ABSTRACT Addressing the complexities of classifying distinct object classes in computer vision presents several challenges, including effectively capturing features such as color, form, and tissue size for each class, correlating class vulnerabilities, singly capturing features, and predicting class labels accurately. To tackle these issues, we introduce a novel hybrid deep dense learning technique that combines deep transfer learning with a transformer architecture. Our approach utilizes ResNet50, EfficientNetB1, and our proposed ProDense block as the backbone models. By integrating the Vit‐L16 transformer, we can focus on relevant features in mammography and extract high‐value pair features, offering two alternative methods for feature extraction. This allows our model to adaptively shift the region of interest towards the class type in slides. The transformer architecture, particularly Vit‐L16, enhances feature extraction by efficiently capturing long‐range dependencies in the data, enabling the model to better understand the context and relationships between features. This aids in more accurate classification, especially when fine‐tuning pretrained models, as it helps the model adapt to specific characteristics of the target dataset while retaining valuable information learned from the pretraining phase. Furthermore, we employ a stack ensemble technique to leverage both the deep transfer learning model and the ProDense block extension for training extensive features for breast cancer classification. The fine‐tuning process employed by our hybrid model helps refine the dense layers, enhancing classification accuracy. Evaluating our method on the INbreast dataset, we observe a significant improvement in predicting the binary cancer category, outperforming the current state‐of‐the‐art classifier by 98.08% in terms of accuracy.