A comprehensive analysis on machine learning based methods for lung cancer level classification

Lung cancer is a major public health issue worldwide that requires early detection using robust techniques. This work initiates a comprehensive review of the use of machine learning (ML) and deep learning-based transfer learning methods for accurate classification of lung cancer stages. A careful analysis is performed to overcome overfitting problems in model performance, considering the minimum child weight and learning rate. A set of machine learning (ML) models including XGBoost (XGB), Light Gradient-Boosting Machine (LGBM), Adaboost, logistic regression (LR), decision tree (DT), random forest (RF), Categorical Boosting (CatBoost) and k-nearest neighbor (k-NN) are systematically implemented and compared. Furthermore, the correlation between features and targets is investigated using a deep neural network (DNN) model, thus characterizing their ability to recognize complex patterns. It is argued that several ML models can be capable of classifying lung cancer stages with high accuracy. Despite the complexity of DNN architectures, traditional ML models such as XGBoost, LGBM, and logistic regression outperformed with superior performance. These models outperformed others in lung cancer prediction across a full set of comparative metrics such as precision, accuracy, recall, and F-1 score. We also found that deep learning-based models performed worse than traditional machine learning approaches on this dataset, with the highest accuracy reported by these models being approximately 0.94. • Traditional machine learning models outperform deep learning in lung cancer staging. • XGBoost and Logistic Regression achieved nearly 100% classification accuracy. • Careful tuning of learning rate and child weight minimized overfitting risk. • Deep learning models underperformed due to limited dataset size and complexity. • Findings support cost-effective, interpretable ML models for clinical lung cancer diagnosis.