Global Contrastive Learning with High-Quality Data in Large Vision-Language Models for Pathological Question Answering

Pathological question answering (PQA) is vital in computational pathology, as it involves interpreting pathological images and answering questions posed by humans. This interaction offers an effective means of engaging with users and enhancing the understanding of pathology-related information. Recent methods developed using large vision-language models (LVLMs), such as QUILT-LLAVA, have made significant progress in advancing PQA. However, existing models, such as those using the QUILT-1M dataset, neglect the quality of the training set during the fine-tuning stage, leading to sub-optimal performance. We recognize that high-quality training data can significantly enhance model performance. Therefore, we design a model-based data filtering strategy to remove images with obvious impurities from the instruction fine-tuning dataset. The filtered high-quality images are then used to fine-tune the model. Additionally, most existing vision-language alignment strategies focus primarily on aligning local features through next-word prediction, leading to a relatively homogeneous granularity in inter-modal alignment. To address this issue, we propose a global-wise alignment module, which introduces global-level contrastive learning during the pretraining stage to establish multi-granularity alignment between pathological images and language descriptions. Based on the above two processes, we design our method, named Global Contrastive Learning with High-Quality Data (GCL-HQD) for pathological question answering in LVLMs. Extensive experiments on two types of experimental settings demonstrate the effectiveness of the GCL-HQD method.