Automated Chest Diseases Classification Using Multimodal Deep Learning

Background: Computer-Aided Diagnosis systems can support physicians by reducing workload, medical errors, and time required for patient care. However, many existing systems primarily rely on medical images while overlooking valuable information available in associated medical reports. This study proposes a multimodal deep learning approach for chest disease classification by integrating chest X-ray images and radiology reports from the MIMIC-CXR dataset. (2) Methods: Different Deep Learning models are evaluated for each modality, followed by a comprehensive comparison of intermediate and late fusion strategies to investigate the effectiveness of combining visual and textual information. The proposed framework also addresses practical challenges associated with multimodal medical data, including generating disease labels for unannotated data, handling class imbalance, aggregating multiple images at the study level, selecting an appropriate radiology report section for textual input, and managing missing textual information. Furthermore, unlike existing fusion strategies that rely on traditional fusion strategies or combine features into a single representation, the proposed label-aware intermediate fusion employs disease-specific queries to guide multimodal feature fusion, generating disease-specific representations for predicting the 14 disease labels. (3) Results: The best-performing approach combines visual features extracted from chest X-ray images using the Swin Transformer with textual features extracted from the Findings section of radiology reports using Bio-Clinical BERT through a label-aware intermediate fusion mechanism. This approach achieved a macro-AUC of 90.7% and a micro-AUC of 94.4%, demonstrating its effectiveness in chest disease classification. (4) Conclusions: The results demonstrate the value of combining complementary visual and textual information for chest disease classification and highlight the importance of selecting an appropriate fusion strategy for multimodal medical data. Beyond its diagnostic contribution, the study illustrates how a biological principle—hierarchical integration of complementary sensory channels, with compensation under modality loss—can be operationalized and evaluated within a clinical learning architecture. Ultimately, these efforts aim to improve, develop, and deliver healthcare services in an integrated manner.

Authors

Institutions

Publication Details

Journal
Biomimetics
Published
2026-11-19
DOI
https://doi.org/10.3390/biomimetics11110842
Primary Topic
Multimodal Machine Learning Applications
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Automated Chest Diseases Classification Using Multimodal Deep Learning

Hachémi Bennaceur, Fahman Saeed, Fatimah Almatrafi
Biomimetics
Multimodal Machine Learning Applications
article

Automated Chest Diseases Classification Using Multimodal Deep Learning

Hachémi Bennaceur, Fahman Saeed, Fatimah Almatrafi
article en

Abstract

Background: Computer-Aided Diagnosis systems can support physicians by reducing workload, medical errors, and time required for patient care. However, many existing systems primarily rely on medical images while overlooking valuable information available in associated medical reports. This study proposes a multimodal deep learning approach for chest disease classification by integrating chest X-ray images and radiology reports from the MIMIC-CXR dataset. (2) Methods: Different Deep Learning models are evaluated for each modality, followed by a comprehensive comparison of intermediate and late fusion strategies to investigate the effectiveness of combining visual and textual information. The proposed framework also addresses practical challenges associated with multimodal medical data, including generating disease labels for unannotated data, handling class imbalance, aggregating multiple images at the study level, selecting an appropriate radiology report section for textual input, and managing missing textual information. Furthermore, unlike existing fusion strategies that rely on traditional fusion strategies or combine features into a single representation, the proposed label-aware intermediate fusion employs disease-specific queries to guide multimodal feature fusion, generating disease-specific representations for predicting the 14 disease labels. (3) Results: The best-performing approach combines visual features extracted from chest X-ray images using the Swin Transformer with textual features extracted from the Findings section of radiology reports using Bio-Clinical BERT through a label-aware intermediate fusion mechanism. This approach achieved a macro-AUC of 90.7% and a micro-AUC of 94.4%, demonstrating its effectiveness in chest disease classification. (4) Conclusions: The results demonstrate the value of combining complementary visual and textual information for chest disease classification and highlight the importance of selecting an appropriate fusion strategy for multimodal medical data. Beyond its diagnostic contribution, the study illustrates how a biological principle—hierarchical integration of complementary sensory channels, with compensation under modality loss—can be operationalized and evaluated within a clinical learning architecture. Ultimately, these efforts aim to improve, develop, and deliver healthcare services in an integrated manner.

BiomimeticsVol. 11(11)
Imam Mohammad ibn Saud Islamic University (SA)
Multimodal Machine Learning Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon