Image Captioning Using Topic Faster R-CNN-LSTM Networks

Image captioning is an important task in cross-modal research in numerous applications. Image captioning aims to capture the semantic content of an image and express it in a linguistically and contextually appropriate sentence. However, existing models mostly trend to focus on a topic generated by the most conspicuous foreground objects. Thus, other topics in the image are often ignored. To address these limitations, we propose a model that can generate richer semantic content and more diverse captions. The proposed model can capture not only main topics using coarse-grained objects but also finds fine-grained visual information from background or minor foreground objects. Our image captioning system combines the ResNet, LSTM, and topic feature models. The ResNet model extracts fine-grained image features and enriches the description of objects. The LSTM model provides a longer context for semantics, increasing the fluency and semantic completeness of the generated sentences. The topic model determines multiple topics based on the image and text content. The topics provide directions for the model to generate different sentences. We evaluate our model on the MSCOCO dataset. The results show that compared with other models, our model achieves a certain improvement in higher-order BLEU scores and a significant improvement in CIDEr score.