The Feasibility of One-Stage Instance Segmentation for Detecting Oral Potentially Malignant Disorders in White-Light Clinical Photographs: A Proof-of-Concept Study

Objectives: Oral potentially malignant disorders (OPMDs) carry a variable risk of malignant transformation, making early detection important. Deep learning relies on specialized imaging, which is often inaccessible in routine practice, and detection from standard clinical photographs remains poorly characterized. We assessed the feasibility of automated OPMD detection from white-light intraoral photographs, compared two convolutional neural network paradigms (global classification with DenseNet-121 versus one-stage instance segmentation with YOLOv8), and identified the main barriers to clinical translation. Methods: A dataset of 1500 photographs (750 OPMD and 750 non-OPMD) from institutional archives and publicly accessible sources was split in an 80:20 ratio for training and testing. Lesion boundaries were annotated by three trainees using Cytomine and validated by three specialists. The DenseNet-121 and YOLOv8-large-segmentation models were evaluated for accuracy, sensitivity, specificity, precision, F1 score, and Wilson 95% CI. Results: DenseNet-121 required extensive manual lesion cropping to converge, negating the automation rationale. YOLOv8-large-segmentation reached 62.2% accuracy (95% CI 56.4 to 67.6), 75% sensitivity (95% CI 67.3 to 81.4), 59.7% precision (95% CI 52.4 to 66.5), 49.3% specificity (95% CI 41.3 to 57.4), and an F1 score of 66.5%, detecting lesions in 96% of the test set. High sensitivity was obtained at a low confidence threshold of 0.1, with correspondingly reduced specificity, and the model produced interpretable color-coded segmentation masks. Conclusions: One-stage instance segmentation is the more viable direction and yields spatially interpretable output, but performance at this dataset scale is not yet clinically sufficient. Dataset scale, threshold calibration, low specificity, and absent patient metadata are the key barriers to address.