HC-MFT: a hierarchical cross-modal fusion transformer for robust and explainable thoracic disease diagnosis

Chest radiography (CXR) and computed tomography (CT) provide complementary views of thoracic disease, yet most automated diagnostic models use a single modality. We present HC-MFT, a hierarchical cross-modal fusion transformer that encodes radiographs with a Vision Transformer and CT slices with an EfficientNet-B3–Transformer hybrid, exchanges information between the two token sequences through stacked bidirectional cross-attention, and combines them with a sample-adaptive, dimension-wise gate. Because no public dataset offers patient-matched CXR–CT examinations for the four classes studied, we evaluate the method on class-matched pseudo-pairs built from the COVID-19 Radiography Database and the SARS-CoV-2 CT-Scan Dataset; both image pools are split into training, validation and test partitions before pairing and audited for near-duplicates, and the CT branch is supervised only with the binary label it carries. With an identical training protocol for all models, HC-MFT reached 96.3% accuracy (95% CI [94.7, 97.7]), a macro AUC of 0.989 and a macro F1 of 0.963 on 600 test pairs covering COVID-19, lung opacity, viral pneumonia and normal radiographs, and a macro F1 of 0.957 ± 0.004 across 5 random splits. HC-MFT was the most accurate and most stable model, but its advantage over the strongest fusion baselines was not statistically significant. Pairing controls showed that the trained model relies almost entirely on the radiograph, and a quantitative saliency analysis revealed substantial attention outside the lung fields, which suggests source-related shortcuts. HC-MFT is therefore a competitive computational fusion framework; clinical use requires validation on genuinely paired, multi-centre cohorts

Authors

Institutions

Publication Details

Journal
Scientific Reports
Published
2026-11-20
DOI
https://doi.org/10.1038/s41598-026-81110-y
Primary Topic
COVID-19 diagnosis using AI
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

HC-MFT: a hierarchical cross-modal fusion transformer for robust and explainable thoracic disease diagnosis

Ibrar Hussain, Khalil Ullah, Sergei Romanov, Shah Khalid et al.
3 links
Scientific Reports
COVID-19 diagnosis using AI
article

HC-MFT: a hierarchical cross-modal fusion transformer for robust and explainable thoracic disease diagnosis

Ibrar Hussain, Khalil Ullah, Sergei Romanov, Shah Khalid, Shah Imran, Yury Markin, Evgeny Karpulevich
article en

Abstract

Chest radiography (CXR) and computed tomography (CT) provide complementary views of thoracic disease, yet most automated diagnostic models use a single modality. We present HC-MFT, a hierarchical cross-modal fusion transformer that encodes radiographs with a Vision Transformer and CT slices with an EfficientNet-B3–Transformer hybrid, exchanges information between the two token sequences through stacked bidirectional cross-attention, and combines them with a sample-adaptive, dimension-wise gate. Because no public dataset offers patient-matched CXR–CT examinations for the four classes studied, we evaluate the method on class-matched pseudo-pairs built from the COVID-19 Radiography Database and the SARS-CoV-2 CT-Scan Dataset; both image pools are split into training, validation and test partitions before pairing and audited for near-duplicates, and the CT branch is supervised only with the binary label it carries. With an identical training protocol for all models, HC-MFT reached 96.3% accuracy (95% CI [94.7, 97.7]), a macro AUC of 0.989 and a macro F1 of 0.963 on 600 test pairs covering COVID-19, lung opacity, viral pneumonia and normal radiographs, and a macro F1 of 0.957 ± 0.004 across 5 random splits. HC-MFT was the most accurate and most stable model, but its advantage over the strongest fusion baselines was not statistically significant. Pairing controls showed that the trained model relies almost entirely on the radiograph, and a quantitative saliency analysis revealed substantial attention outside the lung fields, which suggests source-related shortcuts. HC-MFT is therefore a competitive computational fusion framework; clinical use requires validation on genuinely paired, multi-centre cohorts

Scientific Reports
Russian Academy of Sciences (RU), Federal Almazov North-West Medical Research Centre (RU), Saint Petersburg State Electrotechnical University (RU), Shaheed Benazir Bhutto University, Sheringal (PK), University of Malakand (PK)
Good health and well-being
COVID-19 diagnosis using AI
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon