Predicting Post-Thoracic Surgery Life Expectancy for Lung Cancer Using Machine Learning with SHapley Additive exPlanations

Around 11.6% of all cancer patients around the world suffers from lung cancer. It is still one of the most significant health concerns all over the world. Thoracic surgery is an effective and common treatment for lung cancer. Even though it is considered a medium risk procedure, predicting life expectancy after thoracic surgery is important. In this study, we want to use machine learning for accurately predicting the life expectancy of patients following thoracic surgery for lung cancer. The dataset used in this research has 470 datapoints with 16 attributes. Only 70 of the datapoints are of positive class. We have performed thorough comparison between multiple models which are: K-Neighbors, Random Forest, XGboost, Adaboost classifiers, and an Ensemble Model (Stacking Technique). Since, the dataset was heavily imbalanced we solved it utilizing oversampling technique such as Random oversampling and SMOTE-ENN (Synthetic Minority Oversampling-Edited Nearest Neighbor). SelectKBest method was utilized for feature selection. For validation k-fold method has been used. The optimal model turns out to be Stacking Classifier, which achieved an impressive accuracy of 99.38% (std: 0.0040) using K-fold cross-validation (k=5) when using Random Oversampling. The model mechanism has been explained using SHapley Additive exPlanations (SHAP).