Self-Supervised Multimodal Learning for Tumor Classification in Chest Radiography

Chest radiography plays a crucial role in the early detection and classification of tumors, yet the reliance on labeled data poses significant challenges in medical imaging. To address this, we introduce a Self-Supervised Multimodal Learning for Tumor Classification(SML-TC) framework that effectively combines visual and textual modalities. This approach harnesses both radiographic images and their corresponding reports, allowing the model to develop rich feature representations w/o extensive labeled datasets. By employing self-supervised learning, our method enhances diagnostic performance while overcoming the limitations posed by sparse annotations. Specifically, our framework learns meaningful representations and incorporates data augmentation, even with a limited amount of annotated data, allowing the model to better capture distinctive identifying features for tumor classification. Comprehensive experiments conducted on various chest radiographs reveal that our approach substantially exceeds the performance of conventional supervised learning methods. Additionally, the results demonstrate heightened accuracy in tumor detection and classification, emphasizing the potential impact of our model in clinical practice and its capability to assist radiologists with informed, data-driven interpretations.