A Hybrid Deep Learning Model with Self-Attention for the Classification of Lung Cancer Using Histopathology Image

Lung cancer remains a prevalent health burden and is one of the leading causes of cancer mortality worldwide. Its high mortality rate is partly attributable to histological heterogeneity and the difficulty of detecting it at early stages. An accurate distinction of lung cancer subtypes in histopathological images is crucial for improving the accuracy of diagnosis and planning an appropriate treatment to improve the quality of life of patients. This study proposes a hybrid deep-learning model for classifying cancer types using histopathology images. The ConvNeXt-Tiny is an extension of the ResNet-50 base architecture. This architecture is inspired by both models and introduces self-attention layers to improve both feature extraction and classification performance, leading to a unique model design. The proposed model and two other deep learning models were trained and tested using the public Lung and Colon Cancer Histopathological Image (LC25000) dataset and a private clinical dataset, and their effectiveness was evaluated. The proposed model outperformed the best classification accuracy among the other architectures (98.73% public and 93.17% private), outperforming baseline models, such as ConvNeXt-Tiny (96.27% public and 89.33% private) and ResNet-50 (94.00% public and 87.67% private). The results confirm the robustness and generalization ability of the proposed architecture.