Multimodal transformers for joint diagnosis and retrieval from chest X-rays and radiology reports

Radiological diagnosis frequently hinges on the joint interpretation of medical images and their corresponding narrative reports. However, most existing automated approaches treat these modalities in isolation, thereby overlooking their rich semantic interplay—an essential element for robust clinical understanding. We introduce RadFuse , a unified multimodal transformer framework that concurrently analyzes chest radiographs and associated free-text radiology reports. The model integrates a Vision Transformer (ViT) for visual representation learning and ClinicalBERT for processing clinical narratives. These modalities are fused through stacked self-attention and bidirectional cross-attention mechanisms, enabling fine-grained semantic alignment. RadFuse is optimized via a composite objective that combines binary cross-entropy for multi-label disease classification and a contrastive InfoNCE loss for image-text retrieval. We evaluate RadFuse on two benchmark datasets— MIMIC-CXR and IU X-ray —across two key tasks: disease classification and cross-modal retrieval. On MIMIC-CXR, RadFuse achieves an AUC of 0.847 and a Recall@10 of 78.2 , outperforming prior unimodal and shallow multimodal baselines. Ablation studies demonstrate the critical contribution of cross-attention and contrastive training to the model’s performance. RadFuse provides a scalable and interpretable solution for vision-language integration in clinical settings. Its unified architecture enables multi-task learning, facilitates robust semantic grounding, and sets a new benchmark for multimodal modeling in radiology AI.