Wavelet STC-TransUNet++: Enhanced lung cancer cell segmentation and classification in histopathological images

Lung cancer diagnosis is crucial for improving the survival rates of cancer patients. Nowadays, histopathological image analysis is being used to study cancer cell patterns at the microscopic level to avoid misclassification. Therefore, there is a high demand for accurate segmentation of cancer cell boundaries and correct classification according to their subtypes. To address this need, a novel method named Wavelet STC-TransUNet++ is proposed that leverages the advantages of discrete wavelet transform, selective kernel modules, Vision Transformer and channel attention, integrating them into the U-Net++ architecture. In the proposed architecture, the encoder phase uses discrete wavelet transform to capture approximated frequency components, while the selective kernel module adaptively selects the most relevant features. The bottleneck phase incorporates the Vision Transformer to capture the spatial relationships between cell structures. In the decoder phase, channel attention plays a significant role in reconstructing and refining the feature maps, which, through convolution refinement layers, assist the segmentation head in producing accurate maps. This information is also utilized by the classification layer to correctly classify the lung cancer subtypes adenocarcinoma, squamous cell carcinoma or normal lung tissue. To enable the model to train on a larger, balanced dataset, synthetic data are generated using StyleGAN. The proposed methodology achieves an accuracy of 99.7 %, precision of 99.33 %, and F1-score of 99.35 %, demonstrating that the combination of wavelet-based feature extraction, transformer-based contextual learning, attentional decoder refinement and synthetic data augmentation positions this model as a comprehensive solution for diagnostic accuracy in lung cancer classification.