HC-MFT: a hierarchical cross-modal fusion transformer for robust and explainable thoracic disease diagnosis
Chest radiography (CXR) and computed tomography (CT) provide complementary views of thoracic disease, yet most automated diagnostic models use a single modality. We present HC-MFT, a hierarchical cross-modal fusion transformer that encodes radiographs with a Vision Transformer and CT slices with an EfficientNet-B3–Transformer hybrid, exchanges information between the two token sequences through stacked bidirectional cross-attention, and combines them with a sample-adaptive, dimension-wise gate. Because no public dataset offers patient-matched CXR–CT examinations for the four classes studied, we evaluate the method on class-matched pseudo-pairs built from the COVID-19 Radiography Database and the SARS-CoV-2 CT-Scan Dataset; both image pools are split into training, validation and test partitions before pairing and audited for near-duplicates, and the CT branch is supervised only with the binary label it carries. With an identical training protocol for all models, HC-MFT reached 96.3% accuracy (95% CI [94.7, 97.7]), a macro AUC of 0.989 and a macro F1 of 0.963 on 600 test pairs covering COVID-19, lung opacity, viral pneumonia and normal radiographs, and a macro F1 of 0.957 ± 0.004 across 5 random splits. HC-MFT was the most accurate and most stable model, but its advantage over the strongest fusion baselines was not statistically significant. Pairing controls showed that the trained model relies almost entirely on the radiograph, and a quantitative saliency analysis revealed substantial attention outside the lung fields, which suggests source-related shortcuts. HC-MFT is therefore a competitive computational fusion framework; clinical use requires validation on genuinely paired, multi-centre cohorts
Authors
- Ibrar Hussain (ORCID: https://orcid.org/0009-0004-7212-2354)
- Khalil Ullah (ORCID: https://orcid.org/0000-0001-7265-7325)
- Sergei Romanov
- Shah Khalid
- Shah Imran
- Yury Markin
- Evgeny Karpulevich
Institutions
- Russian Academy of Sciences (RU)
- Federal Almazov North-West Medical Research Centre (RU)
- Saint Petersburg State Electrotechnical University (RU)
- Shaheed Benazir Bhutto University, Sheringal (PK)
- University of Malakand (PK)
Publication Details
- Journal
- Scientific Reports
- Published
- 2026-11-20
- DOI
- https://doi.org/10.1038/s41598-026-81110-y
- Primary Topic
- COVID-19 diagnosis using AI
- Type
- article
- Field-Weighted Citation Impact
- 0.00