Attention-based Multimodal Fusion with Adversarial Network for In-Hospital Mortality Prediction

It is crucial to predict the in-hospital mortality for improving clinical decision-making and optimizing hospital resource allocation. Many recent studies have attempted to integrate multimodal data, such as digital time series and clinical notes from electronic health records (EHRs), to improve the performance of mortality prediction. Although current methods show good performance, it is often difficult to effectively capture fine-grained inter-modal correspondences. In addition, the different distributions and heterogeneous properties of the various modalities lead to modality gaps that severely affect the effectiveness of modal fusion. To overcome these limitations, we propose an Attention-based Multimodal Fusion with Adversarial network (AMFA) for in-hospital mortality prediction. In AMFA, feature extraction is first performed to obtain modality-specific features, and then an attention mechanism is employed to capture inter-modal interrelationships. The main difference with existing methods is that AMFA achieves this by calculating the relative importance of individual features focusing attention on the most relevant features and eliminating irrelevant features, and then reassigning attention between relevant features to obtain finer semantic relevance. Subsequently, we introduce a discriminator network that addresses the modality gap by adjusting the distribution of various modal representations through adversarial training. We evaluate AMFA on two publicly available datasets, and experimental results show that AMFA outperforms several state-of-the-art models in the task of in-hospital mortality prediction.