Colon cancer disease diagnosis based on coati optimization algorithm and modified VGG-Net-CNN

Colon cancer remains a critical global health issue, necessitating accurate and timely diagnosis. This study proposes a hybrid approach that combines a modified VGGNet-based convolutional neural network (CNN) for feature extraction with Coati Optimization Algorithm (COA) for optimal feature selection, improving the classification of histopathological images. Unlike traditional methods relying on principal component analysis or handcrafted features, binary COA enables efficient and automated selection of discriminative features. The approach employs transfer learning with pre-trained CNNs—including VGGNet, AlexNet, and SqueezeNet—fine-tuned on colon cancer datasets. Classification is performed employing multiple machine learning algorithms such as Support Vector Machine (SVM), Decision Tree, Naïve Bayes, Logistic Regression, Ensemble methods, and K-Nearest Neighbors (KNN). Experiments on the LC25000 dataset containing 10,000 images evenly split between cancerous and non-cancerous samples, utilized histogram equalization and data augmentation for preprocessing. A 5-fold cross-validation scheme ensured robust evaluation. Results show that the Decision Tree with VGGNet-COA achieves 98.84 % accuracy and F1-score, KNN with SqueezeNet-COA reaches 98.68 % sensitivity, and the Gradient Boosting ensemble attains the highest accuracy of 98.95 %. Statistical analyses via paired t -test (t = 4.92, p < 0.05) and ANOVA (F ≈ 1500, p < 0.01) confirmed significant differences among models. The proposed CNN-COA framework outperforms conventional methods, demonstrating the potential of integrating deep learning and nature-inspired optimization for reliable, automated colon cancer diagnosis, supporting clinical decision-making and improved patient outcomes.