Deep learning-based automatic segmentation of oral squamous cell carcinoma in histopathological images: a comprehensive evaluation and performance analysis

Introduction: Oral squamous cell carcinoma (OSCC) is a prevalent form of oral cancer, demanding precise and timely diagnosis for improved patient outcomes.Traditional histopathological analysis is labor-intensive and subject to variability, prompting the need for automated and accurate diagnostic tools.Objectives: This study aimed to develop and evaluate a deep learning (DL) algorithm for segmentation of OSCC in histopathological images.Material and methods: A DL-based approach utilizing U2-Net architecture was implemented for semantic segmentation of OSCC.Dataset was split into three parts: 85% for training, 10% for validation, and 5% for testing, with augmentation methods applied to improve model robustness.A batch size of 4 was used during the model training process, image dimensions of 512 1,024 pixels, and optimized with Adam optimizer.Performance metrics included Dice similarity coefficient (DSC), intersection over union (IoU), precision, and recall.Results: Our algorithm achieved a high accuracy of 95.3% in classifying OSCC pixels.DSC was 0.947 and IoU reached 0.902, indicating strong segmentation performance.Validation metrics confirmed robust generalization with a DSC of 0.862 and IoU of 0.77.In the test dataset, the DSC was 0.865 and the F1 score was 0.889, reflecting balanced precision and recall.Validation accuracy was 85.4%, with a Dice coefficient of 0.843.Conclusions: The DL model demonstrated exceptional performance in segmenting OSCC in histopathological images, suggesting significant clinical utility in aiding pathologists with OSCC diagnosis and treatment planning.These findings highlight the potential for integrating AI-based tools into clinical workflows to enhance diagnostic accuracy and efficiency.