Constructing Negative Sample Pairs to Mitigate Language Priors in Medical Visual Question Answering

Objective: Our research aims to design a new debiased model that utilizes Constructing Negative Sample Pairs (CNSP) to address the language priors problem in Medical Visual Question Answering (Med-VQA). Language priors can cause models to rely on linguistic cues rather than image content, potentially leading to misdiagnosis and inappropriate treatment recommendations in clinical decision-making. Methods: First, to avoid the language priors caused by the poor feature extraction ability and multimodal feature fusion effect of the model, we adopt a two-stage model of the pre-training stage and fine-tuning stage. In the pre-training stage, we combine Masked Image Modeling (MIM), Masked Language Modeling (MLM), and Image-Text Matching (ITM) pre-training tasks. Second, to solve the language priors caused by the imbalance of the dataset and the loss function design problem, we construct negative sample pairs through self-supervised learning to balance the number of positive and negative sample types and back-propagate the loss value obtained through the negative sample pairs to the fine-tuning stage model to curb the generation of language priors. Finally, to verify the transferability of our method, we added our method to other Med-VQA models and achieved significant improvement in results. Results: Our method significantly enhances the model's accuracy, achieving 78.7%, 84.3%, and 81.3% on the public datasets VQA-RAD, SLAKE, and VQA-MED-2019, respectively. Conclusion: Our CNSP model provides a novel and effective solution to the problem of language priors in Med-VQA, and as a plug-and-play approach, can be applied to other Med-VQA models.