A Color-Aware Unsupervised Segmentation Network for Nuclei in Histopathology Images

Precise nuclei segmentation is essential in digital pathology, yet it faces difficulties due to the wide existence and morphological diversity of nuclei, staining protocols, and imaging conditions. The process of annotating nuclei instances demands skilled pathologists to carefully trace the outlines of the nuclear structures, which is both labor-intensive and expensive. Consequently, deep learning-based segmentation techniques, which heavily depend on large datasets for training, face challenges in maximizing their effectiveness in this domain. Here our study introduces an unsupervised method for segmenting nuclei instances in whole slide histopathology images. Our approach is divided into two primary phases: self-supervised training and clustering-based segmentation. In the first phase, mask pre-training allows the encoding network to capture the essential color features of pathology images, enabling it to extract more comprehensive and detailed information about the nuclei. In the second stage, the encoder backbone that has been trained is used to derive clustering features from the images. The network learning and clustering processes are optimized jointly, and the final results of instance segmentation are achieved by means of subsequent post-processing techniques. The results indicate that the proposed method exhibits enhanced performance compared to other unsupervised nuclei segmentation approaches.