Enhancing Lung Cancer Classification Performance on Diverse Datasets: Utilizing Advanced Feature Selection and Novel Ensemble Approaches

Lung cancer is a severe and often fatal disease characterized by uncontrolled cell growth in lung tissues. The high death rate associated with lung cancer presents a significant public health problem. In this study, our objectives are to develop novel models for lung cancer prediction using a diverse set of datasets, improving the outcomes of earlier studies on the problem. we addressed the critical issue of imbalanced datasets in lung cancer research by employing techniques such as undersampling and Synthetic Minority Over-sampling Technique (SMOTE) to achieve balanced datasets. We focused on feature selection techniques such as correlation based analysis, feature importance and coefficient based analysis to make efficient models. Different machine learning techniques have been employed, including decision trees, random forests, K-nearest neighbor, Gradient Boosting Machine, AdaBoost, XGBoost, LightGBM, CatBoost, logistic regression classifiers, deep neural networks, and a curriculum learning-based voting classifier. Among all the models evaluated, Random Forest and LightGBM demonstrated the highest accuracy, achieving a performance rate of 97%. Subsequently, we proposed a model using a voting classifier which uses multiple base learners such as Random Forest, GBM, XGBoost, Light GBM, CatBoost, and Deep Neural Network. This ensemble approach also achieved an accuracy rate of 97%. In conclusion, high-performance models for lung cancer prediction were developed, which could potentially help in the improvement of non-invasive lung cancer prediction models in medical settings.