Classification of cervical cells from the Pap smear image using the RES_DCGAN data augmentation and ResNet50V2 with self-attention architecture

Abstract Cervical cancer is a type of cancer in which abnormal cell growth occurs on the surface lining of the cervix. In this study, we propose a novel residual deep convolutional generative adversarial network (RES_DCGAN) for data augmentation and ResNet50V2 self-attention method to classify cervical cells, to improve the generalizability and performance of the model. The proposed method involves adding residual blocks in the generator of the DCGAN to enhance data flow and generate higher-quality images. Subsequently, a self-attention mechanism is incorporated at the top of the pre-trained models to allow the model to focus more on significant features of the input data. To evaluate our approach, we utilized the Pomeranian and SIPaKMeD cervical cell imaging datasets. The results demonstrate superior performance, achieving an accuracy of 98% with Xception and 96.4% with ResNet50V2 on the Pomeranian dataset. Additionally, DenseNet121 with self-attention achieved accuracies of 92% and 95% in multiclass and binary classification, respectively, using the SIPaKMeD dataset. In conclusion, our RES_DCGAN-based data augmentation and pre-trained with self-attention model yields a promising result in the classification of cervical cancer cells.