Multi Head Attention-based Visual Content Analysis

The purpose of image captioning (IC) is to provide an accurate textual description of images. Traditional image captioning models are based on advanced signal processing and convolutional neural networks. However, there is a demand for the captioning models to pay higher attention to the images to provide richer, or more dense, textual descriptions of the images. In this paper, an image captioning model with Multi-Head Attention (MHA) is proposed. Here, image features extracted using EfficientNetB0 are enhanced using the Squeeze and Excitation network. Further, features are applied to a modified transformer-based encoder-decoder to generate captions. The obtained captions are refined using a refinement network based on T5-architecture, thus providing grammatically correct captions. The proposed approach achieves a BLEU-4 score of 24.83 on the Flickr8K dataset.