NUCSEG: An Effective Detectron2 Architecture for Nuclei Instance Segmentation in Histological Images

Nuclei are often minuscule objects within histological whole slide imaging (WSI), presenting a labor-intensive task for pathologists to identify and analyze their characteristics. The precise identification of nuclei is essential in cancer pathology to conduct both quantitative and qualitative assessments of cancerous cells found within a slice of a tumor in order to assess the level of malignancy of the tumor; and to predict the outcome of the tumor. Manual identification is labor-intensive and prone to errors, whereas automatic recognition is complicated by the varied appearances and overlapping characteristics of nuclei, along with artifacts from digital photography. To address this challenge, a deep learning model called the NUCSEG model is suggested for automatic detection, which helps pathologists distinguish instances of nuclear objects within the WSI images. The NUCSEG model, inspired by the Detectron2 framework, utilizes Mask R-CNN with various enhanced ResNet backbone networks and BiFPN to accurately detect the nuclei boundaries and overlapping instances at multiple resolution levels to effectively identify the boundaries between the nucleus and cytoplasm. The experimental results show that the best NUCSEG configuration using the deep ResNet with a 101-layer backbone achieved a performance with a BBOX AP50 and AP75 of 80.141% and 55.278%, respectively. Furthermore, NUCSEG outperformed YOLOv11, particularly demonstrating superior performance at high IoU for 0.75, indicating its ability to propose regions and localize regions with consistent, high confidence ratings, even when nuclei overlap or have irregular shapes.