Personalized Clinical Risk Stratification Based on Multimodal Medical Data Fusion

A representation of the patient capturing physiological data, contextual data, and radiological information is required for personalized clinical risk stratification. In this paper, we propose a multi-modal approach that combines structured information extracted from electronic health records, unstructured information from clinical notes, and chest X-rays collected from the MIMIC-IV and MIMIC-CXR-JPG databases within the first 24 hours in intensive care. We encode structured information using TabTransformer, unstructured text with BioClinicalBERT, and radiology images with DenseNet121. These modality-specific representations are then mapped to a shared latent space and fused by a patient-specific gated attention mechanism to predict in-hospital mortality and classify into low, intermediate, and high-risk groups. The fused representation gave an AUROC of 0.912 ± 0.006, an AUPRC of 0.478 ± 0.014, and a Brier score of 0.074 ± 0.003, outperforming concatenation of the features, which scored an AUROC of 0.889 ± 0.007. When structured, text, and radiological information were dropped, the model's discrimination performance declined to 0.853 ± 0.009, 0.881 ± 0.008, and 0.894 ± 0.007, respectively. The mortality rates among the risk-stratified groups ranged from 1.9 ± 0.4% to 28.7 ± 1.6%, indicating clear clinical separation.