Breast cancer detection redefined: Integrating Xception and EfficientNet-B5 for superior mammography imaging

• Autoencoder-CNN Fusion: Precision Boost in Breast Cancer Diagnosis. • Autoencoder Driven Image Quality Enhancement. • Multi-Head Attention for Precise Tumor Segmentation. • Thorough Evaluation: Diverse Metrics for Segmentation and Classification. Breast cancer is a major cause of female mortality, with early detection crucial for effective treatment. This study aims to enhance mammographic breast cancer detection by integrating an autoencoder’s encoder unit with advanced Convolutional Neural Networks (CNNs), incorporating soft attention gates and feature merging for improved accuracy. A custom multi-head attention mechanism is utilized for precise tumor segmentation, with a focus on robust validation across diverse datasets. The research involved 3,816 mammogram samples (2,376 benign, 1,440 malignant) and employed deep learning techniques combining model fusion and autoencoders for classification. A custom multi-head attention mechanism was applied for tumor segmentation. The models were validated on publicly available datasets, MIAS and CBIS-DDSM. On the MIAS dataset, Xception and EfficientNet-B5 CNN models outperformed others, achieving a classification accuracy of 96.88% after autoencoder integration. For segmentation, the model demonstrated strong alignment with tumor regions, achieving a Dice Coefficient of 0.4353, Intersection over Union (IoU) of 0.2998, and F1-Score of 0.4318. This study developed a robust deep learning approach combining Xception and EfficientNet-B5 for breast cancer diagnosis and segmentation. The fused model demonstrated high classification accuracy and reliable segmentation performance, indicating strong potential for clinical applications in early breast cancer detection and treatment planning.