Segmentation-Based Attention with Spatial Encoding for Image Captioning

With the proliferation and widespread deployment of cameras, image captioning for automatically and concisely generating natural language descriptions of image contents has attracted a lot of attentions. In this work, we propose a method for image captioning based on segmentation attention. The segmentation masks of the objects are from Mask R-CNN. The proposed segmentation attention performs the attention in object mask level and further encodes the spatial information. The makes the generated caption more accurate. The experiment exhibits that the proposed method can achieve the high BLUE scores in Flickr 8K dataset.