Assessing Effective Token Length of Multimodal Models for Text-to-Image Retrieval

Multimodal embedding models have been widely adopted in text-toimage retrieval, enabling direct comparison between text and image modalities.However, how well they handle long text is poorly understood.For instance, Long-CLIP found that OpenAI's CLIP model, despite having a 77-token input limit, maintains optimal performance for only 20 tokens-its effective token length.In this paper, we build on the Long-CLIP study, and extend the analysis to other widely used multimodal models and find their effective token length.Unlike Long-CLIP, we examine how domain-specific language influences changes in effective token length and explore its implications on different domains.Based on our findings, we create a comprehensive reference of various models' effective token length across different domains; offering deeper insights into the true limitations of multimodal models used in text-to-image retrieval.Finally, we introduce a systematic benchmark that determines the effective token length of any multimodal model using a given dataset.Our results show that the effective token length is consistently lower than the input token limit for all models, meaning that these models cannot utilize all the text that can be given to them.We also find that the effective token length varies by dataset, with domain-specific language influencing how much text a model can use before retrieval performance plateaus.Our code is available for reproducibility at https://github.com/aiforsec/EffectiveTokenLength-MModels