Robust oriented object detection for posterior teeth in mixed dentition

The mixed dentition stage presents significant diagnostic challenges in panoramic radiography due to the high structural density and spatial superposition of primary roots and developing permanent germs. Traditional horizontal bounding box (HBB) detection methods suffer from a “geometric mismatch,” failing to accurately localize tilted teeth in crowded dentitions. This study aims to develop and validate a high-precision oriented object detection (OBB) framework to resolve these localization ambiguities. A retrospective dual-center study was conducted utilizing 1,148 panoramic radiographs. A specialized dataset annotating primary molars, developing premolars, and first permanent molars was constructed. The YOLOv11-OBB architecture was trained on a primary cohort (n = 1,040). Model performance was evaluated on an internal test set (n = 104) and an independent external validation cohort (n = 108) to assess generalizability. The YOLOv11-OBB-Large model achieved exceptional performance, yielding a Mean Average Precision (mAP@50–95) of 0.904 on the internal dataset and maintaining robust accuracy (mAP@50–95 = 0.896) on the external dataset. The Nano model demonstrated a superior efficiency-accuracy trade-off, achieving an inference latency of 141ms on a standard CPU (vs. 958ms for Large) with minimal performance loss (mAP@50–95 = 0.879). In the stratified analysis by tooth position, the model exhibited consistent reliability across both permanent and primary dentitions. Mandibular premolars and molars achieved the highest precision (mAP@50–95 > 0.94), while primary molars maintained robust detection rates (mAP@50–95 > 0.86) despite the complexities of root resorption and germ overlap. The proposed OBB framework effectively addresses the geometric limitations of traditional detectors, providing precise localization and rotation information essential for downstream tasks such as automated space analysis and eruption monitoring. The model demonstrates strong generalizability across different clinical centers. To foster reproducibility and facilitate future research, the source code and pre-trained weights are publicly available at https://github.com/Zheng-Yunhao/Mixed-Dentition-OBB-Detection.