Classification of histopathological images for oral cancer in early stages using a deep learning approach

Abstract Problem – Diagnosing early-stage oral squamous cell carcinoma (OSCC) is necessary for patient treatment and survival. The staining normalization and augmentation of histopathological images can play an important role in improving the accuracy of early oral cancer (OC) detection, especially with imbalanced and limited datasets, which is a difficult manual task because it is time-consuming, requires effort, and is subject to variation among different pathologists. Applying these methods can enhance the performance of deep learning models used for OSCC detection. Aim – This study addresses the aforementioned challenges by showing how the random stain normalization and augmentation (RandStainNA) technique can be applied to datasets as a preprocessing step along with transfer learning models to categorize histopathological images of OC into three classes. Methods – The performance of three models – ResNet-50, VGG-19, and an ensemble model – in image classification is compared. Using fivefold cross-validation for training, our framework compares two main tasks. The best results were achieved on the task in which the models were pretrained using patches, and the RandStainNA technique was applied to the images as a preprocessing step. We trained the models to extract features from the relevant images and then determine the final classifications. Results – Our proposed framework achieved the best results in the early diagnosis of OC using the NDB-UFES dataset and the ResNet-50 model, demonstrating 73.33% balanced accuracy, 74.68% precision, and 74.35% recall, as well as a 92.21% area under the curve. Conclusion – Our proposed framework was effective in improving the accuracy of early OC detection, especially when working with small and imbalanced datasets. In addition, this method may contribute to enhancing the generalizability of models and using them in diverse laboratories.