A reproducible and generalizable framework for breast cancer diagnosis from mammographic Imaging: Integrating radiomics and attention-based deep features

This study aimed to develop a reliable diagnostic framework for breast cancer detection using 3D tomosynthesis mammography. It integrated radiomic features with deep learning-derived features from an attention-based 3D autoencoder to enhance diagnostic accuracy, improve feature consistency across scanners, and increase interpretability for clinical use. The dataset consisted of 6672 cases categorized into normal, benign, and malignant groups. Radiomic features were extracted and harmonized using ComBat to address variability between scanners. Deep features were derived using an attention-based 3D autoencoder, focusing on diagnostically significant regions. Feature reproducibility was assessed using intra-class correlation coefficients (ICC), retaining only those with ICC >0.85. To optimize the feature space, dimensionality reduction and feature selection methods such as Principal Component Analysis (PCA), Least Absolute Shrinkage and Selection Operator (LASSO), and Recursive Feature Elimination (RFE) were applied. Machine learning models—including XGBoost, CatBoost, Random Forest, and a stacking ensemble—were trained with a stratified 70/30 train-test split and evaluated using 5-fold cross-validation. Performance metrics, including the Area Under the Curve (AUC), accuracy, and sensitivity, were used to assess the models. The stacking model combined with RFE demonstrated the best performance, achieving an AUC of 95.75 %, an accuracy of 94.29 %, and a sensitivity of 94.74 %. XGBoost with PCA also performed strongly, with an AUC of 94.40 %. The attention-based autoencoder improved the quality of deep features significantly, achieving a testing AUC of 92.32 %, an accuracy of 91.22 %, and a sensitivity of 90.12 %, outperforming the non-attention model. Combining radiomic and deep features yielded better results than using either feature set individually. This study presents a robust and interpretable diagnostic framework for breast cancer detection by effectively combining radiomic and attention-enhanced deep features. The proposed approach outperforms existing models by addressing critical challenges related to feature reproducibility and clinical interpretability. Future work should focus on integrating multimodal data and optimizing the framework for real-time clinical applications.