Bootstrapping Radiography Pre-training via Siamese Masked Vision-Language Modeling with Complementary Self-distillation

Diagnosing thoracic diseases from chest X-rays (CXR) using deep learning faces unique challenges due to the high anatomical similarity across images and the critical nature of minute anomalies. In this paper, we introduce a novel self-supervised learning framework, Siamese Masked Vision-Language Modeling with Complementary Self-distillation (SMVLM), designed to enhance disease diagnosis in CXR by addressing these specific challenges. First, we tackle the problem of high anatomical similarity and subtle variance in CXR by employing a complementary masking strategy in a Siamese network setup, which forces the model to focus on subtle, yet diagnostically relevant features from both global and local perspectives. Secondly, we enrich our model’s learning capabilities by integrating a dual masked image-text contrastive loss that aligns radiographic findings with their corresponding text report, harnessing the synergistic potential of multimodal data. In conjunction with this, cross-modal image-text pre-reconstruction with registers is introduced to deepen the contextual understanding of the CXR images and radiology report, ensuring a comprehensive feature representation. Extensive evaluations on benchmark datasets demonstrate that our method significantly outperforms existing approaches, providing a robust solution for the accurate and reliable diagnosis of thoracic diseases from CXR images.