ToothLens: ROI-guided entropy-adaptive and boundary-aware enhancement for tooth instance segmentation in panoramic radiographs

Accurate segmentation of individual teeth in the Orthopantomogram radiograph is a crucial task in dentistry. The technical challenges stemming from OPG image acquisition properties, including background noise, inter-ROI contrast variability, local intensity, non-stationarity, and detailed anatomical boundaries, are of significant research interest. Existing image processing and deep learning segmentation approaches are primarily focused on designing segmentation frameworks for higher accuracy, without addressing limitations in OPG imaging through appropriate pre-enhancement strategies. This paper presents a novel ROI-adaptive hybrid enhancement framework, a focused tooth enhancement methodology solely designed by analysing imaging challenges of OPG. The designed framework is a pre-processing enhancement pipeline for OPG instance segmentation using Mask R-CNN with a ResNet-50-FPN backbone. The framework comprises four sequential modules: ROI decomposition, Entropy-driven adaptive CLAHE, ROI-constrained FFT, and regularized edge refinement. The framework is trained on the OPG dataset acquired from AB Shetty Memorial Institute of Dental Science (ABSMIDS), with five-fold cross-validation and data augmentation to ensure model generalization. The proposed full segmentation pipeline achieves an IoU of 0.8105, a Dice coefficient of 0.8953, a precision of 0.9133, and recall of 0.8780 under deployable conditions using a leakage-free protocol in which ROI masks are used for enhancement is always generated from the predicted masks. Ground-truth-mask upper-bound analysis illustrates better IoU and Dice values (p < 0.001), demonstrating an upper-bound reference for enhancement potential. The ROI-mask-quality sensitivity analysis illustrates current deployable performance is directly related to ROI localization accuracy. A detailed enhancement framework analysis is performed using PSNR, SSIM, MSE, and MAE, demonstrating improved structural enhancement. Comparative analysis demonstrates that ROI-FFT outperforms Global FFT, encouraging region-based enhancement. The observed inverse relationship between segmentation and image fidelity metrics (PSNR, SSIM) underscores the importance of task-specific structural enhancement over pixel-level fidelity optimization in deep learning and traditional image-processing segmentation pipelines for OPG analysis.