Hybrid multimodal heart failure predicting technique based on structural EHR and sequential ECG signals

Heart failure (HF) relies one of the most significant global health challenge, which required an accurate and timely prediction to save patients lives. This paper presents a hybrid multi-modal predicting technique via integrating both longitudinal structured electronic health records (EHRs) information and sequential electrocardiogram (ECG) signals in order to obtain a better heart failure possibility prediction. Through combining both clinical tabular data capturing from patients medical histories with dynamic sequential ECG signals. In this paper a multi-modal heart failure forecasting system (HFFS) is proposed. Where two models are overlapped, the gated recurrent unit with decay model (GRU-D) and residual network (ResNet), so, the GRU-D is effective for capturing the temporal progressions of patients’ health pointer. While ResNet is used to avoid vanishing gradients in neural networks, it is used for learning complex patterns from medical data, to increase the efficiency of the model.This paper concentrates mainly on the potential of utilizing multi-modal data fusion in medical support systems. Through utilizing the customized datasets, the proposed multi-modal evaluation accuracy obtained is 97.89%, which clearly offering a 4.5% improvement over separate models as a standalone.