Study-level cross-modal retrieval of chest x-ray images and reports with adapter-based fine-tuning

Cross-modal retrieval is crucial for improving clinical decision-making and report generation. However, current technologies mainly focus on linking single images with reports, ignoring the need to comprehensively observe multiple images in real clinical environments. Additionally, differences in imaging equipment, scanning parameters, geographic regions, and reporting styles in chest x-rays and reports cause inconsistent data distributions, which challenge model reliability and generalization. To address these challenges, we propose a study-level cross-modal retrieval task for chest x-rays and reports to better meet clinical needs. Our study-level approach involves cross-modal retrieval between multiple images and reports from patient exams. Given a set of study-level images or reports, our method retrieves relevant reports or images from a database, providing a more realistic reflection of clinical scenarios compared to traditional methods that link single images with reports. Furthermore, we introduce an adapter-based pre-training and fine-tuning method to enhance model generalization across diverse data distributions. Through comprehensive experiments, we demonstrate the advantages of our method in pre-training and fine-tuning. In the pre-training stage, we compare our method with the latest techniques, showing the effectiveness of integrating study-level image features using a vision transformer and aligning them with report features. In the fine-tuning stage, we compare the adapter-based fine-tuning method with the latest methods of full-parameter fine-tuning and conduct ablation studies with common head-based and full-parameter fine-tuning methods, proving our method's efficiency and significant potential for practical clinical applications. This study proposes a study-level cross-modal retrieval task for matching chest x-ray images and reports. By employing a pre-training and fine-tuning strategy with adapter modules, it addresses the issue of data distribution inconsistency and improves retrieval performance.