MSAE-DL: enhancing breast cancer classification through hybrid self-attention integration, feature fusion, and ensemble classification in digital breast tomosynthesis

Abstract Breast cancer is a significant global health challenge, with highest rates of occurrence and mortality worldwide. Early detection is important for the improvement of patient outcomes and the reduction of the overall burden of the disease. Digital breast tomosynthesis (DBT) scans offer three-dimensional images of the breast tissue and is becoming a valuable tool in the detection of breast abnormalities. However, accurately classifying DBT scans is challenging due to the complexity of the anatomy of the breast and the presence of minor abnormalities. This study introduces the MSAE-DL system for the multi-class classification of DBT scans. The system incorporates a novel multi-head self-attention model with a unique ensemble classification model. Features were extracted from the Mod_AlexNet Self-Attention model and fused with histogram of oriented gradients (HOG) descriptors. Subsequently, feature vectors are reduced using three feature selection models. Finally, a novel ensemble classification model is introduced and fuses class and classifier weights for the final prediction using various classifiers. The system demonstrates optimal performance in classifying DBT scans into normal, benign, and malignant classes, achieving an accuracy of 90.13%, precision of 92.77%, and f1-score of 91.03%. The experimental results underscore the potential of this approach in enhancing DBT classification into three different classes, rather than simply binary classification.