A Multimodal Deep Learning Approach for Pre-Operative Tumor-Node-Metastasis Staging in Oral Cancer Using Computed Tomography Imaging and Pathological Slide Data

Oral cancer is one of the most prevalent cancers globally, characterized by high rates of recurrence and metastasis, making early diagnosis crucial for improving patient survival. Accurate preoperative tumor-node-metastasis (TNM) staging is essential for determining effective treatment plans and surgical strategies. Although pathological examination remains the gold standard for TNM staging of oral cancer, imaging data can complement pathology, helping clinicians make more precise preoperative assessments and addressing the limitations of single-modality approaches. In this study, we propose a novel multimodal model for oral cancer staging that integrates preoperative CT images and whole-slide imaging (WSI). To overcome the heterogeneity between CT and WSI features, especially the lack of interaction between macrolevel CT features and microlevel pathological features, we introduce a CT-guided collaborative attention module. Specifically, CT features serve as queries, while patches of WSI are treated as key-value pairs. The corresponding keys are computed through a fully connected layer with trainable weights. The model architecture includes two separate pathways for feature extraction: CT features are extracted using a U-Net-based network, while WSI features are extracted with a multi-instance network utilizing a hierarchical attention mechanism. The CT-guided collaborative attention module facilitates interaction and fusion between these two modalities, resulting in a unified feature representation. This fused feature is then passed through a fully connected layer to produce the final TNM staging prediction for oral cancer. By combining both macro- and microlevel features, our model addresses the limitations of traditional single-modality methods, enabling more accurate tumor boundary delineation. Compared to existing techniques, our multimodal approach improves diagnostic accuracy and provides more reliable staging results. This advancement has the potential to significantly enhance early diagnosis and treatment strategies for oral cancer, offering a more comprehensive and precise method for staging the disease.