Multimodal Deep Learning and Knowledge Graph Framework for Early Disease Screening

Abstract — Traditional artificial intelligence (AI) systems often treat medical images and clinical data separately. However, screening various diseases can facilitate their early detection and prompt clinical action. Decoupled handling of medical images and clinical data could potentially limit AI’s ability to associate patients’ unique risks, abnormal image findings, symptoms, lab results, etc. In this paper, we propose a multimodal deep learning and knowledge graph framework that integrates patients’ chest x‐ray images and relevant structured clinical data from the electronic health records and screens five common thoracic conditions, i.e., atelectasis, cardiomegaly, pulmonary edema, pleural effusion, and pneumonia. Specifically, we plan to employ transfer learning‐based models as the backbones to extract informative embeddings of patients’ chest radiographs, a multilayer perceptron to learn structured clinical data embeddings, and a graph neural network to model patients’ relevant biomedical knowledge embeddings. Then, we propose an availability‐aware gated‐fusion module to integrate embeddings from all modalities and predict patients’ risks of target diseases. Our framework is intended not only to predict probabilities for each disease but also to label patients’ disease risks as low, moderate, or high risk, thereby assisting clinicians’ disease screening and decision‐making, rather than making clinical decisions on its own. We plan extensive experiments by integrating two credentialed-access benchmark datasets, MIMIC‐CXR‐JPG and MIMIC‐IV, with the CheXpert dataset for external image‐based evaluation. Our experimental setup accounts for patient‐level data splitting, time leakage prevention, and uncertainty‐aware labels, and is designed to evaluate the performance of the proposed framework in terms of class‐balanced training strategies, ablation study, calibration analysis, subgroup analysis, missing modalities analysis, etc