Enhancing pap smear image classification: integrating transfer learning and attention mechanisms for improved detection of cervical abnormalities

Cervical cancer remains a major global health challenge, accounting for significant morbidity and mortality among women. Early detection through screening, such as Pap smear tests, is crucial for effective treatment and improved patient outcomes. However, traditional manual analysis of Pap smear images is labor-intensive, subject to human error, and requires extensive expertise. To address these challenges, automated approaches using deep learning techniques have been increasingly explored, offering the potential for enhanced diagnostic accuracy and efficiency. This research focuses on improving cervical cancer detection from Pap smear images using advanced deep-learning techniques. Specifically, we aim to enhance classification performance by leveraging Transfer Learning (TL) combined with an attention mechanism, supplemented by effective preprocessing techniques. Our preprocessing pipeline includes image normalization, resizing, and the application of Histogram of Oriented Gradients (HOG), all of which contribute to better feature extraction and improved model performance. The dataset used in this study is the Mendeley Liquid-Based Cytology (LBC) dataset, which provides a comprehensive collection of cervical cytology images annotated by expert cytopathologists. Initial experiments with the ResNet model on raw data yielded an accuracy of 63.95%. However, by applying our preprocessing techniques and integrating an attention mechanism, the accuracy of the ResNet model increased dramatically to 96.74%. Further, the Xception model, known for its superior feature extraction capabilities, achieved the best performance with an accuracy of 98.95%, along with high precision (0.97), recall (0.99), and F1-Score (0.98) on preprocessed data with an attention mechanism. These results underscore the effectiveness of combining preprocessing techniques, TL, and attention mechanisms to significantly enhance the performance of automated cervical cancer detection systems. Our findings demonstrate the potential of these advanced techniques to provide reliable, accurate, and efficient diagnostic tools, which could greatly benefit clinical practice and improve patient outcomes in cervical cancer screening.