Multimodal Retrieval Method for Images and Diagnostic Reports Using Cross-Attention

Background: Conventional medical image retrieval methods treat images and text as independent embeddings, limiting their ability to fully utilize the complementary information from both modalities. This separation often results in suboptimal retrieval performance, as the intricate relationships between images and text remain underexplored. Methods: To address this limitation, we propose a novel retrieval method that integrates medical image and text embeddings using a cross-attention mechanism. Our approach creates a unified representation by directly modeling the interactions between the two modalities, significantly enhancing retrieval accuracy. Results: Built upon the pre-trained BioMedCLIP model, our method outperforms existing techniques across multiple metrics, achieving the highest mean Average Precision (mAP) on the MIMIC-CXR dataset. Conclusions: These results highlight the effectiveness of our method in advancing multimodal medical image retrieval and set the stage for further innovation in the field.