Advancing Lung Cancer Prediction: A Comparative Study of ML, DL & Hybrid Models

One of the deadliest diseases that causes millions of deaths every year is Lung cancer, where early and accurate detection can play an important role in increasing the survival rate. This research presents a comparative analysis of Machine Learning, Deep Learning and a hybrid approach for lung cancer prediction by effectively using the dataset obtained from the trusted source Kaggle, where it consists of several features that are crucial in lung cancer prediction. Three ML models like XGBoost, Random Forest and Support Vector Machine were used in this research and among them XGBoost ranked the highest with an accuracy of 88.50%. Whereas, Deep Learning models such as FullyConnectedNN, CNN1D, and TabTransformer showed promising results, where FullyConnectedNN achieved the highest accuracy of 91.3%. Additionally, hybrid models integrating XGBoost and FCNN achieved the accuracy of 89.20% which is over ML models but it was not able to outperform FCNN, so overall FCNN performed the best among all other approaches. Visual representation like Exploratory data analysis, feature importance and perturbation testing were implemented and analyzed. The results obtained indicates that DL models efficiently captured feature interactions which led to the effectiveness in predictive performance as compared to the ML & hybrid techniques. Our research conducted a strong comparison between ML, DL & hybrid approaches precisely on a structured dataset for lung cancer prediction. While previous studies have also done something similar to this but our study evaluates the models not only through performance metrics but also with robustness analysis because this analysis can effectively tell the total number of changed predictions for a model, that plays a major role in evaluating a model. Our study highlights DL’s ability to obtain better predictive performance and also to capture complex patterns than conventional ML & hybrid approaches.