Vision language models for scientific image analysis: an evaluation highlighting opportunities and challenges

Recent advancements in vision language models (VLMs) have opened new avenues for analyzing complex visual data. Models such as ChatGPT, Gemini, Llama and LLaVA have gained prominence for their ability to process both visual and textual data, excelling in tasks like natural image captioning, visual question answering (VQA), and reasoning. Similarly, the Segment Anything Model (SAM) by Meta has demonstrated remarkable segmentation capabilities. Given the importance of microscopy images in fields like biology, medicine, and materials science—where visual data is often analyzed alongside textual information from captions, reports, or literature—it is critical to evaluate the effectiveness of these models on such data. This study assesses the capabilities of ChatGPT-5, Gemini-2.5Pro, Llama-3.2V, LLaVA-1.5 and SAM-2 on classification, segmentation, counting, and VQA tasks using microscopy images. ChatGPT and Gemini excelled in comprehending microscopy images, while SAM performed well in object isolation. Although their performance falls short of domain expert accuracy, particularly when faced with complexities such as impurities, overlaps, and irrelevant artifacts, these models show clear gains compared to prior versions. These findings highlight the promise of VLMs in scientific image analysis and the need for further advancements to meet the demands of expert-level tasks.