A deep learning model for multiclass lung cancer classification using multimodal data fusion

Lung cancer is the most prevalent malignant tumour in terms of morbidity and mortality. Accurate classification and risk staging are essential for developing appropriate treatment plans and achieving precision in lung cancer diagnosis and treatment. A lung cancer pathological image classification model was trained using ResNet152. The image input structure of ResNet152 was modified to support custom input image block sizes. A multimodal deep learning model (MMLM) was developed that integrated features extracted from the trained ResNet152 (RNet), standardized RNA sequencing data, methylation microarray data, and clinical information through a feature-level fusion method. This process emulates the way in which pathologists integrate multimodal information to make decisions. The incorporation of Class Activation Mapping (CAM) and UMAP mapping enhanced the interpretability of the model, thus increasing its transparency and fostering greater trust in its decision-making process. In the context of lung cancer classification on an independent test set, compared with monomodality RNet, the MMLM demonstrated superior performance. The areas under the curve (AUC) values of the MMLM were 0.999, 1.000, and 0.980, which significantly surpassed RNet’s values of 0.822, 0.732, and 0.781, respectively. Similarly, the average precision (AP) values of the MMLM were 0.999, 1.000, and 0.909, which surpassed those of RNet (0.793, 0.606, and 0.471). In addition, supplementary validation of the ability of the MMLM to predict lung cancer risk staging yielded encouraging results. The MMLM integrates pathological images, RNA sequencing data, methylation data, and clinical information to improve lung cancer classification and prognosis. However, challenges remain, such as limited access to comprehensive clinical data, patient privacy concerns, and computational resource demands. The MMLM holds significant potential to become a powerful tool for personalized lung cancer diagnosis and treatment with continued advancements in data integration and privacy.