Mitigating Text Dominance in Medical Vision–Language Learning for Long-Tailed Multi-Label Chest X-Ray Classification

Background/Objectives: Medical vision–language learning integrates radiographic images and textual information for chest X-ray (CXR) classification. However, in long-tailed multi-label CXR classification, modality imbalance can cause models to rely disproportionately on radiology reports while underutilizing visual information. This study investigates text dominance and proposes a modality-balanced framework to strengthen visual representation while preserving multimodal performance. Methods: Experiments used MIMIC-CXR with patient-level training, validation, and test splits of 32,942, 4024, and 3955 paired samples, respectively. Swin-Tiny and PubMedBERT served as visual and textual encoders. The proposed framework combines adaptive gated fusion, Asymmetric Loss (ASL), and training-time modality dropout. It was compared with Binary Cross-Entropy (BCE) concatenation, ASL concatenation, gated fusion, and cross-attention fusion. Results: The BCE baseline achieved a Macro AUROC of 0.9957 under Image + Text evaluation and 0.5804 under Image-Only evaluation, while Text-Only evaluation yielded 0.9955, indicating substantial modality disparity. Other alternatives achieved Image-Only Macro AUROC values of 0.5610–0.5720. The proposed framework improved Image-Only Macro AUROC to 0.7697 and Macro PR-AUC from 0.1814 to 0.3428 while maintaining an Image + Text Macro AUROC of 0.9958. The modality-dominance gap decreased from 0.4151 to 0.2260, a 45.6% reduction in performance disparity. Grad-CAM and Integrated Gradients provided insights into visual evidence and textual attribution, revealing reliance on explicit disease-related terminology in reports. Conclusions: These findings suggest that combining adaptive gated fusion, ASL, and modality dropout can improve Image-Only performance while preserving multimodal discrimination in long-tailed multi-label CXR classification.