Stroke Outcome Prediction via Multi-level Feature and Multi-modal Fusion Network

Deep learning has been widely applied in medical fields, including medical image analysis, disease prediction, drug development, and clinical decision-making. In disease prediction, researchers have developed advanced models to enhance accuracy. However, medical data comes in various forms, such as tables and images, and few models effectively leverage both modalities for prediction, particularly in stroke outcomes. This paper focuses on predicting outcomes for ischemic stroke and proposes a deep learning model that utilizes multi-level image features and multi-modal feature fusion for disease prediction. The model is designed as a multi-task learning framework that simultaneously addresses image segmentation and disease prediction tasks. The segmentation task employs a U-shaped network, with an encoder capable of extracting multi-level image features from various encoder blocks. These multi-level features are fused through a multi-level feature fusion (MFF) module, providing more direct information about lesions for subsequent multi-modal feature fusion. For the prediction task, clinical features are extracted from tabular data and then integrated with image features via a multi-modal fusion (MMF) module. The MMF module employs an attention mechanism to comprehensively integrate medical image features with each variable in the clinical data. The framework is trained with segmentation and classification losses for joint supervision, enabling the model to segment lesion locations while simultaneously performing disease prediction. Extensive experimental results demonstrate that our framework can achieve the best prediction performance.