Beyond Human Vision: The Role of Large Vision Language Models in Microscope Image Analysis

Vision language models (VLMs) such as LLaVA, ChatGPT-4, and Gemini have recently emerged and gained the spotlight for their ability to comprehend the dual modality of image and textual data showing impressive performance on tasks such as natural image captioning, visual question answering, and spatial reasoning. Additionally, a universal segmentation model by Meta AI, Segment Anything Model (SAM) shows unprecedented performance at isolating objects from unforeseen images. Because medical experts, biologists, and materials scientists routinely examine microscopy or medical images in conjunction with textual information in the form of captions, literature, or reports, and draw conclusions of great importance and merit, it is essential to evaluate their performance on these images. In this study, we charge ChatGPT, LLaVA, Gemini, and SAM quantitatively with classification, segmentation and counting tasks. We observed that ChatGPT and Gemini were impressively able to comprehend the visual features in microscopy images, while SAM was quite capable at isolating artifacts in a general sense. However, the performance was not close to that of a domain expert – the models were readily encumbered by the introduction of impurities, defects, object overlaps and diversity present in the images.