A hybrid deep learning model for mammographic breast cancer detection: Multi-autoencoder and attention mechanisms

This study aims to develop a robust diagnostic framework for breast cancer detection in mammographic images by integrating multi-autoencoder-based feature extraction with attention mechanisms. The objective is to address key limitations in traditional and state-of-the-art methods, including limited adaptability, manual feature dependency, and lack of interpretability, ensuring enhanced diagnostic accuracy and clinical utility. This study utilizes a multi-center dataset of 5987 mammograms (malignant: 36 %, benign: 31.9 %, normal: 32.1 %). Images were standardized to 256 × 256 pixels, with intensity normalization and augmentation. A multi-autoencoder framework with six independently pre-trained autoencoders extracted diagnostic features. Recursive Feature Elimination (RFE) with XGBoost was applied for feature selection. Attention mechanisms prioritized diagnostically significant regions. Classification performance was evaluated using accuracy, sensitivity, specificity, F1-score, and AUC-ROC, while segmentation was assessed using IoU, Dice score, and localization accuracy. Five-fold cross-validation ensured robustness, and Adam optimizer with early stopping was used for optimal model training. The proposed framework demonstrates high accuracy in both segmentation and classification for breast cancer detection. Attention-based segmentation achieved 91.5 % localization accuracy, with IoU = 0.87 and a Dice score of 0.89, ensuring precise identification of diagnostic regions. The multi-autoencoder classification model attained 94.2 % sensitivity and 96.4 % AUC in training, with 92.4 % sensitivity and 95.8 % AUC on independent testing, outperforming traditional statistical features. XGBoost surpassed other classifiers, including Random Forest, SVM, and Logistic Regression. These results validate the model's robustness, interpretability, and clinical applicability, establishing an AI-driven diagnostic tool for accurate breast cancer segmentation and classification. The proposed framework advances breast cancer detection by offering high accuracy, adaptability, and interpretability. Future work should explore multimodal imaging integration and lightweight implementations for real-time deployment in clinical environments.