Optimized deep learning ensemble for accurate oral cancer detection using CNNs and metaheuristic tuning

Oral cancer presents a significant worldwide health challenge, necessitating early and accurate diagnosis to enhance survival rates. Conventional diagnostic techniques are frequently manual, laborious, and prone to variability, thus postponing identification and treatment. This research presents an enhanced deep learning ensemble model for the categorization of oral cancer, aimed at improving diagnostic precision and efficiency. The suggested methodology amalgamates Enhanced EfficientNet-B5—supplemented with Squeeze-and-Excitation (SE) and Hybrid Spatial-Channel Attention (HSCA) modules—with ResNet50V2, capitalizing on their synergistic advantages in precise lesion identification and profound hierarchical feature extraction. Hyperparameter optimization was conducted with the Tunicate Swarm Algorithm (TSA) to enhance convergence rate and mitigate overfitting. The ensemble model, trained on the ORCHID dataset of high-resolution histopathology pictures, attained a classification accuracy of 0.99 following TSA optimization, surpassing individual CNNs and conventional models that generally exhibit lower accuracies (about 0.95–0.98). Significant enhancements were observed in precision, recall, F1-score, along with a considerable decrease in false positives. The results highlight the efficacy of the proposed model as a viable AI-assisted diagnostic tool for the early diagnosis of oral cancer, providing a significant improvement over current clinical methodologies. • An ensemble model integrates EfficientNet-B5 and ResNet50V2 for oral cancer detection • Squeeze-and-Excitation and Hybrid Attention modules enhance feature extraction • Tunicate Swarm Algorithm optimizes hyperparameters for improved model accuracy • Achieved 99% accuracy with significant reduction in false positives • Robust AI framework supports early diagnosis and clinical decision-making