ITRScore: A Multi-Granularity Evaluation Method for Image-Text Retrieval Models Based on LLM

In current cross-modal image-text retrieval evaluation, there are often struggles with capturing fine-grained matches between images and texts using existing methods. This limitation leads to a poor correlation between evaluation results and human judgement. To address this, we propose a novel evaluation method for ITR models called ITRScore. This method simulates the human evaluation process by converting image retrieval results into multi-granularity textual descriptions using image caption model, and then scoring the retrieval results with a large language model. Additionally, we conducted experimental validation on three retrieval models: CLIP, BLIP, and BLIP-2. Compared to traditional methods, ITRScore showed a 60.39% improvement in Kendall's tau correlation coefficient, a 66.79% improvement in Spearman's rank correlation coefficient, and achieved 85.94% and 84.54% consistency with human evaluations in object and attribute capture, respectively.