Exploring counting capabilities of multimodal large language models

This paper investigates the counting capabilities of multimodal large language models (MLLMs) on both textual and visual tasks, using OpenAI's state-of-the-art GPT-4O model as a representative example. We assess the model’s ability to count characters in words and objects in images through simple prompts, finding moderate accuracy: 69.5% for text and 80.3% for images. Performance decreases as input complexity rises, with lower accuracy for longer words and higher object densities in images. These findings suggest that while MLLMs are competent in basic counting tasks, they encounter challenges in more complex scenarios, highlighting areas for future enhancement in counting accuracy across modalities.