DentSurgNet: Automated surgical planning via YOLO-based instance segmentation of dental anomalies in panoramic X-rays

Accurate and efficient instance segmentation of dental structures in panoramic X-rays is critical for enabling real-time AI-assisted surgical planning in digital dentistry, especially in resource-constrained clinical settings where high computational cost limits practical deployment. This study implements a lightweight deep learning framework for automated surgical planning in dentistry by instance segmentation of panoramic X-rays using three YOLO architectures: YOLOv8n-seg, YOLOv11n-seg, and YOLOv12n-seg in nano (n) variants. The goal is to achieve high performance while maintaining low computational cost for deployment in resource-limited clinical environments. A combined dataset of 2,842 panoramic X-ray images with 9,299 annotated instances were curated from Roboflow sources and divided into distinct training (2,030), validation (533), and testing (279) sets. All models were fine-tuned using transfer learning with AdamW optimization over 400 epochs at an image resolution of 640×640 pixels. Performance was evaluated using Precision, Recall, and mean Average Precision (mAP) for both detection boxes and segmentation masks at IoU thresholds of 0.5 and 0.5–0.95. Among the tested models, YOLOv11n-seg achieved the best overall performance with Box mAP@50 = 0.892 and Mask mAP@50 = 0.886, surpassing other YOLO variants while maintaining low computational cost (∼4.9 ms per image). Lightweight YOLO-based segmentation models offer high accuracy and efficiency for multi-class dental anomaly detection, supporting real-time AI-assisted surgical planning in digital dentistry.