Combining CNN and Transformer for Enhancing Medical Image Captioning

Composing radiology reports in hospital settings is labor-intensive, requiring significant expertise from radiologists. This paper introduces a novel approach to enhance medical image captioning by combining Convolutional Neural Networks (CNN) with Transformers. Based on pretrained Inception V3for visual feature extraction, our model employs a Transformer encoder-decoder architecture to generate detailed and contextually accurate descriptions of medical images. Evaluations on the Indiana University chest X-ray dataset demonstrate our model's good performance over traditional methods, as evidenced by BLEU, ROUGE-L, and METEOR scores. This approach shows significant potential in improving diagnostic processes and patient care in medical imaging.