Assessment of uncertainty estimation and distillation for reliable 3D dental mesh segmentation in orthodontics

Abstract The integration of deep learning into computer-aided orthodontic treatment planning has enabled automated tasks such as tooth segmentation from 3D dental models. However, the reliability of these systems remains a concern due to their black-box nature and tendency toward overconfident predictions. In this work, we present a comprehensive evaluation of uncertainty estimation methods for tooth segmentation on 3D intraoral scans. We investigate several state-of-the-art neural network architectures combined with uncertainty estimation techniques including predictive entropy, Monte Carlo dropout, test-time augmentation, deep ensembles, and evidential deep learning. Evaluation on the Teeth3DS benchmark, supplemented with synthetically incomplete scans and a custom out-of-distribution dataset, shows that sampling-based approaches, particularly deep ensembles, provide the most reliable uncertainty estimates. In contrast, entropy-based methods and evidential deep learning often produce overconfident predictions and fail to identify major segmentation errors. We further demonstrate that uncertainty estimation can detect out-of-distribution samples such as scans with artifacts or missing regions. Although deep ensembles provide the most reliable uncertainty estimates, their computational cost limits practical deployment. We therefore investigate uncertainty distillation and show that a single student model can achieve ensemble-like uncertainty quality with only one forward pass.