A hybrid vision transformer–CNN ensemble with radiomics fusion for robust cross-dataset breast cancer detection in mammography

Abstract Breast cancer is among the leading causes of cancer-related mortality in women worldwide. Early detection through mammograms can significantly increase the chances of survival. However, mammogram classification remains a challenging task due to subtle lesion features and variations across patients and datasets. Therefore, classic convolutional neural networks (CNNs) fail to generalize across different mammographic sources. In this paper, the proposed method is a hybrid Vision transformer and CNN ensemble learning framework integrated with feature fusion for robust cross-dataset classification. The framework consists of a dual-branch feature extractor that fuses a deep CNN for local texture extraction with a Vision transformer (ViT) for global context extraction, while simultaneously extracting handcrafted radiomics features (shape, intensity, and texture) from the lesion region. Then, deep and radiomics features are fused and fed to a stacked ensemble learning classifier comprising XGBoost, SVM, and LightGBM, with a logistic regression meta-learner for performance optimization and overfitting reduction.The proposed framework is also robust across datasets, achieving AUC scores of 0.938 and 0.925 when trained on CBIS-DDSM and evaluated on INbreast, and when trained on INbreast and evaluated on CBIS-DDSM, respectively, demonstrating strong generalization under domain-shift conditions. The proposed fusion model demonstrates promising potential for robust and reliable use in clinics. On the combined CBIS-DDSM and INbreast dataset, the proposed framework achieved an average accuracy of 96.3%, average MCC of 0.928, and average AUC of 0.983 across the five validation folds. The best-performing fold achieved a peak accuracy of 96.9%, peak MCC of 0.936, and peak AUC of 0.986.