Graph attention network-based multimodal approach for lung diseases classification

Lung diseases remain one of the major causes of mortality worldwide. Medical experts usually require information from multiple sources, including clinical symptoms, laboratory and pathological tests, in combination with chest X-rays, to confirm a lung disease diagnosis. However, the manual interpretation of information collected from various sources is time-consuming and may lead to misdiagnosis of lung disease due to structural overlap and similar symptoms. An automated classification framework that integrates information from multiple sources, such as clinical notes, disease symptoms, and chest X-ray images, could enhance diagnostic precision and optimize clinical workflow efficiency. Therefore, this study proposed a multimodal approach based on the Graph Attention Network that integrates the pre-trained Clinical ModernBERT and RAD-DINO models, allowing for granular cross-modal interactions and providing robust multimodal representation learning to advance lung disease classification. The experiments are conducted using a publicly available lung disease dataset, which consists of chest X-ray images and their corresponding clinical text. For multiclass lung disease classification, the proposed multimodal approach achieved an accuracy of 95.73%, precision of 95.75%, recall of 95.72%, and F1-score of 95.70%, along with an expected calibration error of 0.0124. The proposed graph attention network-based multimodal framework outperformed existing state-of-the-art methods and effectively integrated imaging and textual information at the token level, offering an accurate and reliable automated alternative for lung disease diagnosis.