Novel Deep Network Architecture for Multicell Segmentation of Pap Smear Images

Cervical cancer remains a major cause of mortality among women, particularly in low- and middle-income countries. Accurate and automated segmentation of Pap smear images is essential for developing reliable AI-assisted diagnostic tools. In this study, we propose CelluVGG19, a novel deep learning architecture that integrates a modified VGG19 backbone with a U-Net framework to achieve precise segmentation of nuclei and cytoplasm in multicell cervical images. The model replaces standard convolutions with separable convolutions to improve computational efficiency and incorporates residual blocks in the decoder to enhance gradient flow and spatial detail recovery. CelluVGG19 was evaluated on the CRIC publicly available dataset and a private dataset from Pomeranian Medical University, achieving segmentation accuracies of up to 99.60% for nuclei and 99.10% for cytoplasm, on the Pomeranian dataset. These results demonstrate the model’s effectiveness in handling complex cellular morphologies and its potential for broader applications in cervical cancer screening.