A Capsule Network-Based Hybrid Model for Lung Nodule Detection

Lung cancer is a leading cause of death, where early detection is one of the only chances of survival. Manual early detection is time-consuming and error-prone, as it relies on the ability of radiologists to detect small nodules on hundreds of Computed Tomography (CT) images. To address the limitations of human analysis, Computer-Aided Detection (CAD) systems based on Convolutional Neural Networks (CNNs) can assist radiologists but require large datasets, struggle with image variations and lack interpretability. A Capsule Network (CapsNet) can offer an alternative, preserving spatial hierarchies and needing fewer training samples. This work presents a three-stage hybrid model combining You Only Look Once (YOLO) and a CapsNet for lung nodule detection and classification. The pipeline includes YOLOv11 for initial detection, a CapsNet for verification, and YOLOv11 for improved detection. The approach is evaluated on a combined LIDC-IDRI and Lung-PET-CT-Dx dataset. Experiments were done to compare the performance of a Full Image CapsNet classifier versus a Cropped Image CapsNet classifier, where only the Region of Interest (ROI) from detected nodules was used for classification. Results indicate that the Cropped Image CapsNet achieves superior performance with a classification accuracy of 95.57%, compared to 93.07% for the Full Image CapsNet. Despite improved precision, the three-stage pipeline shows slightly lower overall detection accuracy (89.98%) than standalone YOLOv11 (90.79%) due to increased False Negatives (FNs). While a CapsNet enhances classification reliability for FNs, further improvements in bounding box generation and a CapsNet architecture are needed to mitigate its higher False Positive (FP) rate. Future research should optimize the CapsNet, explore alternative detection models, and apply multi-class classification to distinguish benign from malignant nodules. The findings of this study contribute to the ongoing efforts in improving automated lung cancer diagnosis, offering a approach that leverages both CNN-based object detection and a CapsNet’s spatial feature encoding capabilities to enhance nodule detection accuracy and reduce radiologist workload.