Contrastive Multimodal Learning for Chest X-Ray Analysis: Leveraging Text and Image Synergy

Chest X-ray analysis has traditionally relied on either visual or textual data, but a comprehensive understanding of patient conditions can benefit from their combination. We introduce Contrastive Multimodal Learning (CML), a framework designed to leverage the interplay between textual reports and X- ray images to improve diagnostic capabilities. By employing a contrastive learning approach, CML effectively captures the relationship between text and image pairs, facilitating a joint embedding space for feature alignment. The architecture includes stages for feature extraction, utilizing convolutional neural networks for image data and transformers for textual data. A contrastive loss function minimizes the distance between correctly matched pairs while increasing it for mismatched ones. Using a diverse dataset comprising various pathologies and their narrative reports, our experiments reveal that CML achieves superior classification accuracy and better generalization on previously unseen data compared to traditional multimodal methods.