Image-to-Text Summarization using DeiT & GPT-2

This work presents a visual summarization framework for a medical image captioning system that leverages a Data-efficient Image Transformer (DeiT) as the visual encoder and GPT-2 as the language decoder within an encoder-decoder architecture. The model is trained and evaluated on the Indiana Chest X-ray dataset to automatically generate radiology summaries from medical images. Our objective is to improve clinical text generation by summarizing visual diagnostic data into concise and semantically meaningful captions. Evaluation using standard Natural Language Processing (NLP) summarization metrics demonstrates the effectiveness of our model. Specifically, our approach achieves a BLEU score of 0.0680, ROUGE-1 of 0.2713, ROUGE-2 of 0.0981, ROUGE-L of 0.2268, and a METEOR score of 0.0940, outperforming a ViT-GPT2 baseline in all major summarization metrics. This work demonstrates the potential of integrating vision transformers with autoregressive language models for medical image summarization, laying a foundation for AI-assisted radiology reporting systems. Keywords— Chest X-ray, Image Captioning, Vision-to-Text summarization, Transformers, DeiT, GPT-2, Medical Imaging and Deep Learning.