Optimizing Lung Cancer Diagnosis Through Dimensionality Reduction and Transfer Learning

Globally, lung cancer is a major contributor to cancer-related deaths, making it one of the most fatal forms of the disease, primarily because diagnoses are often made in advanced stages due to the lack of early symptoms. Detecting diseases at an early stage is vital for increasing survival rates, as it significantly enhances treatment effectiveness and patient outcomes. Advances in diagnostic methods, including low-dose CT scans and machine learning models, are essential for detecting lung cancer earlier and improving survival chances. This study presents a system designed to address the challenge of using CT scan images To ensure prompt and accurate identification of lung cancer, a critical tool for identifying potential tumors. The proposed approach integrates UMAP for dimension reduction, which helps in managing the high-dimensional nature of CT scan images, and leverages transfer learning with fine-tuned EfficientNetB3 to enhance classification accuracy and efficiency. Dataset balancing is achieved through augmentation techniques to address class imbalance, a common issue in medical imaging datasets. Refining the parameters of the EfficientNetB3 model with pre-trained ImageNet weights, the model converges faster and delivers superior results. Additionally, a custom callback mechanism dynamically adjusts the learning rate as per the validation loss trends, aiding in the prevention of overfitting and ensuring effective optimization throughout training. The model records an accuracy level of $\mathbf{9 6. 3 \%}$, indicating its potential for accurate lung cancer detection. These results underscore the effectiveness of combining UMAP for dimension reduction, transfer learning, data balancing, and dynamic learning rate adjustments in overcoming the challenges of lung cancer diagnosis through computed tomography (CT) scan data.