Enhancing Nucleus and Cytoplasm Segmentation in Cervical Cytology Using Deep Learning

Cervical cancer, with over 660,000 cases in 2022, ranks as the fourth most frequent malignancy in women globally, with low- and middle-income nations accounting for approximately 94% of the 350,000 annual deaths. The challenging diagnosis of cervical cancer, often detected at advanced stages, has been addressed by recent advancements in high-throughput quantitative imaging of 2D cell cultures. Image segmentation automates the traditionally manual process of Pap smear cell diagnosis. While conventional approaches are effective, they face limitations in underrepresented cellular regions, noise, and blurriness in microscopy. Semantic segmentation causes inter-class competition, which causes performance issues for popular segmentation algorithms such as U-Net and Mask R-CNN in multi-class contexts. Our novel approach addresses these challenges by treating nucleus and cytoplasm segmentation as separate tasks, significantly enhancing overall segmentation performance. The nucleus, characterized by higher density and darker appearance, presents a more readily learnable target for the U-Net algorithm. While watershed and similar methods partially mitigate the issue of overlapping and occluded cells, challenges persist. A viable approach is to use pre-trained open-source models like Cellpose, which have been trained on thousands of images. Fine tuning these models through transfer learning for cytoplasm segmentation has produced remarkable outcomes. Pairing segmented nuclei and cytoplasm via centroid distance analysis enhances segmentation accuracy and resolves cellular overlaps. With an accuracy of 99.3%, an IoU of 83.91%, a Dice Coefficient of 91.04%, and precision of 89.88% for nucleus segmentation, and 98.81%, 95.62%, and 91.81% for cytoplasm segmentation, our method demonstrated remarkable accuracy across several parameters. These findings show how our method may help with cervical cancer diagnosis and detection.