MVBV: A Multimodal Prediction Model for Lung Cancer Leveraging Cross-Modal Attention and Vision Transformer with CT Scans and Clinical Text

Lung cancer is one of the most significant threats to human health, and early detection is crucial for patient survival and recovery. This paper proposes a multimodal deep learningbased lung cancer prediction model, named the Multimodal Lung Cancer Image Prediction Model with VGG-BERT-ViT (MVBV), aimed at addressing the high misdiagnosis rates, invasive risks, and isolated modality information challenges present in traditional lung cancer diagnostic methods. The model achieves multimodal feature fusion through a three-stage processing pipeline: First, a pre-trained VGG16 convolutional neural network is employed to extract deep visual features from lung CT images. Second, an innovative cross-modal attention mechanism is introduced, where image features serve as queries and BERT-encoded clinical text features act as keys and values, enabling semantic alignment and dynamic weighted fusion of heterogeneous modalities. Finally, a Vision Transformer (ViT) is utilized to capture global contextual dependencies, and a linear classifier outputs malignancy or benign predictions. The experiment was conducted based on the public LIDC-IDRI dataset containing $\mathbf{1, 0 1 8}$ chest CT scans. The model demonstrated significant performance improvements across three core metrics: Accuracy, F1-score, and the Area Under the ROC Curve (AUC). The MVBV model (with ViT configured as Base/16) achieved an Accuracy of 0.912, an F1-score of 0.888, and an AUC value of 0.941. Compared to the single-modal baseline model DenseNet121, this represents an absolute AUC increase of 0.058 (a relative increase of $5.8 \%$). These results validate the effectiveness of the multimodal fusion strategy in enhancing the diagnostic accuracy for lung cancer. This approach overcomes the limitations of single-modality analysis and provides a novel technical pathway for the development of non-invasive and precise early lung cancer screening systems.