Research on multimodal modal fusion application in pulmonary embolism diagnoses

Pulmonary embolism is a life-threatening illness caused by blockages in the pulmonary arteries, usually due to blood clots. This condition requires accurate diagnosis on time in order to prevent critical complications like cardiac arrest or chronic pulmonary hypertension. As a result, this study explores the capability of data fusion in a multimodal scenario to improve diagnostic accuracy in PE through the integration of Electronic Medical Records with imaging data from CT Pulmonary Angiography. Deep learning models such as PENet, ResNet3D_50, ResNext3D_101, and DenseNet were used for the analysis of imaging data. The EMR data were analyzed using logistic regression and other machine learning methods. A Late Fusion approach that combined the outputs from the EMR and imaging models, using averages, resulted in a diagnostic accuracy with an AUC of 0.94. This is a perfect example where combined clinical and imaging features addressed the shortcomings of one-modality approaches. Advanced fusion techniques, such as MISA, were also tried, but simpler strategies often performed better than them when the amount of data was limited. Overall, the results point out the promise of multimodal fusion in advancing PE diagnostics and also take note of the selection of an appropriate strategy for clinical application.