Hybrid ensemble framework for cancer classification: integrating machine learning and deep learning with explainable AI insights

Abstract Cancer classification is one of the important fields of study that plays a critical role in determining accurate diagnosis and effective treatment strategies. This work introduces a hybrid ensemble learning method using radiomics features. It combines machine learning and deep learning methodologies for better classification of lung cancer types and includes explainable AI. The proposed method aims to achieve higher classification performance by combining the strengths of deep learning and machine learning models. Classification performance is increased by combining machine learning algorithms such as Random Forest, Gradient Boosting, Extreme Gradient Boosting, Light Gradient Boosting Machine, Support Vector Machine, and TabNet model as a deep learning model in a hybrid structure. In addition, PNG conversion from DICOM images, extraction of radiomic features from the image, and optimization steps with Optuna are also included. In this study, where three different types of lung cancer were classified as Adenocarcinoma, Small Cell Carcinoma and Squamous Cell Carcinoma, 99% accuracy rate was achieved by using the voting classifier method. Experimental studies show that the proposed hybrid ensemble learning method provides higher accuracy, sensitivity, and specificity than traditional machine learning models and other ensemble approaches. These results reveal that the hybrid ensemble learning approach offers significant potential as an innovative method in cancer classification. The usability of the developed model in medical applications and its applicability to real-world problems will be evaluated in more detail in further studies. This study aims to establish a basis for developing more reliable and practical solutions in the field of cancer classification.