Exploring Multimodal Large Language Models for Medical Image Captioning

Image captioning involves using models that combine methods from Computer Vision(CV) and Natural Language Processing (NLP) to generate textual descriptions of images. Inthe biomedical field, this process is known as Diagnostic Captioning (DC), where modelsautomatically generate diagnostic text from one or more medical images. DC not onlydescribes the images, but also interprets them to provide a diagnosis, helping healthcareprofessionals gain an initial understanding of a patient’s medical conditions. To achievethis, researchers have formulated DC models that can interpret images and generatedescriptions based on specified instructions. This study aims to unlock new possibilities inthe biomedical domain by harnessing the power of Multi-modal Large Language Models(MLLMs) through techniques like task-specific fine-tuning and few-shot learning. It alsoexplores innovative methods to integrate alternative image captions with a language model(LM) that has fewer parameters.