Improving Diagnostic Sensitivity in Imbalanced Oral Cancer Image Classification: A Comparative Study of CNN and Transformer Architectures

Background: Class imbalance remains a major limitation in artificial intelligence based oral cancer diagnosis, particularly in small clinical image datasets where malignant lesions are underrepresented. Methods: Using a dataset of 3000 mobile-acquired oral cavity images categorized into four diagnostic classes (healthy, benign, oral potentially malignant disorders, and oral cancer), we systematically evaluated the impact of imbalance mitigation strategies on diagnostic performance. Three backbone architectures, EfficientNet, Vision Transformer, and Swin Transformer, were trained under four conditions: raw imbalanced training, random under-sampling, random over-sampling, and medical-safe data augmentation. Performance was evaluated using stratified five-fold cross-validation, and all reported metrics correspond to the mean performance across the validation folds. Results: In multiclass evaluation, augmentation yielded the strongest overall performance, with EfficientNet achieving the highest macro-F1 score (0.669 ± 0.024) and area under curve (0.881 ± 0.015). Under clinically oriented binary malignant-risk evaluation (high-risk vs. low-risk lesions), sensitivity reached 0.817 ± 0.030 with EfficientNet, while data augmentation maintained competitive malignant-risk detection across the evaluated architectures. In contrast, random undersampling consistently yielded lower descriptive performance across architectures. Conclusions: These findings indicate that augmentation-based imbalance mitigation is associated with improved multiclass and oral cancer specific performance while maintaining competitive overall discrimination. The results provide practical guidance for developing clinically robust artificial intelligence systems for early oral cancer detection in imbalanced real-world datasets.