A Multi-Modal Contrastive Learning Framework for Identifying Similar Patients with Cervical Dysplasia from Unlabeled Colposcopy Data

Cervical dysplasia, a precancerous condition of the cervix, can progress to cervical cancer if left untreated, making its early identification crucial. The proposed framework utilizes a multi-modal contrastive learning approach to identify similar patients of cervical dysplasia by combining textual doctor's prescriptions and colposcopic images. This novel method stands out by leveraging contrastive learning with multi-modal data, which is effective in scenarios with limited labeled data. The framework integrates deep learning techniques to extract complex features from colposcopic data as well as patients' textual report through utilizing unimodal text and image encoders as base encoders. Then a Convolutional Neural Network (CNN)-based Variational auto encoder (VAE) is added to extract salient features. Modality fusion is achieved by using separate encoders for image and text representation generation, followed by concatenating these representations for training in the multi-modal vector space using a contrastive learning approach. The experimental plan involves data preparation, model training, and inference to identify similar patients with cervical dysplasia. The model aims to capture optimal latent embeddings to identify similar patients accurately from unlabeled dataset. Therefore, by extracting meaningful representations from the combined data modalities, the framework aims to find the similar patients in cervical dysplasia, offering a novel and potentially impactful avenue for healthcare analytics.