Hybrid multi-CNN feature fusion based on gradient vector flow with ant colony optimization for histopathological breast cancer classification

Abstract Breast cancer is a serious disease because malignant tumors can metastasize and threaten life. This study develops a hybrid framework for eight-class subtype recognition and binary benign-versus-malignant histopathological image classification. The framework was evaluated on BreakHis histopathological images at 100× and 200× magnification. Average and Laplacian filtering enhanced image quality before Gradient Vector Flow segmentation isolated pathological regions. ImageNet-pretrained GoogLeNet, ResNet34, and MobileNet served as fixed feature extractors. Global Average Pooling generated compact representations from each network. These representations were evaluated individually and in pairwise and complete fused combinations to examine complementary information across architectures. Ant Colony Optimization selected informative features and reduced each representation’s dimensionality. XGBoost then classified the selected features for eight-class subtype recognition and binary benign-versus-malignant discrimination using consistent experimental settings across both magnification levels. For eight-class classification, the complete GoogLeNet–ResNet34–MobileNet–XGBoost configuration achieved 95.40% accuracy at 100× magnification and 96.50% at 200×. At 200×, it reached 94.78% precision, 95.25% sensitivity, 96.10% AUC, and 99.38% specificity. Binary classification accuracy reached 99.80% at 100× and 99.50% at 200×. Independent BRACS evaluation across benign, atypical, and malignant categories produced average precision, accuracy, sensitivity, AUC, and specificity of 95.67%, 95.80%, 96.10%, 94.43%, and 98.13%, respectively. BRACS remained independent from the BreakHis development data, providing external assessment on a distinct breast histopathology source. Patient-clustered bootstrap analysis quantified sampling uncertainty around the reported accuracy estimates across both magnification levels and tasks. The findings demonstrate strong classification performance across the evaluated tasks. They support combining complementary CNN representations, ACO-based feature selection, and XGBoost while motivating further evaluation across additional independent histopathology datasets.