Optimizing Multimodal Transformers for Medical Image Captioning: Enhancing Automated Descriptions via AI Systems

In contemporary diagnostic workflows, medical image captioning has emerged as a pivotal advancement, combining deep learning methodologies and transformer architectures to enhance accuracy and efficiency in medical interpretations. This paper proposes the optimization of multimodal transformers for automated medical image captioning, focusing on integrating Vision Transformers (ViT) and Bidirectional Auto-Regressive Transformers (BART) with novel variations such as Swin Transformers and GPT-2. We use a robust multimodal AI framework to explore how these architectures synergize to generate coherent and diagnostically relevant captions for radiological images. We assess the performance of multiple transformer models by employing the ROCO dataset, containing paired X-ray images and expert-generated reports. Our findings demonstrate that the ViT + BART combination yields the most stable and accurate captions, minimizing training and validation loss. In contrast, DEiT + MBART displayed instability, highlighting the need for further hyperparameter tuning. Through this comparative analysis, we underscore the critical role of transformer-based models in reducing the cognitive load on medical professionals, enhancing diagnostic accuracy, and promoting real-time, automated medical image interpretation.