Effective Preprocessing and Explainability for Better Lung Cancer Prediction

The application of artificial intelligence (AI) in the medical field has expanded significantly in recent years. Among its various subsets, machine learning (ML) demonstrates immense potential for enhancing the diagnosis and treatment of diseases. In this study, we aim to enhance the classification outcomes of recent lung cancer research by effective preprocessing techniques and emphasizing the impact of features derived from the ‘Survey Lung Cancer’ dataset. To improve classifier performance, different tasks such as SMOTE (Synthetic Minority Over-sampling Technique) for handling class imbalance and MinMaxScaler for feature scaling, were applied. These preprocessing steps significantly enhanced model accuracy. Specifically, the SVM classifier achieved a PR AUC of 99%, while the Random Forest (RF) classifier demonstrated the best performance across other evaluation metrics, achieving an accuracy of 97.91 %. Additionally, SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations) were employed to identify the most influential features contributing to each model's predictions. These techniques helped identify the most influential features contributing to each model's predictions, enhancing interpretability and understanding of their outputs.