Incorporating Transformer-based Scene Graphs Progressively for Image Captioning

Image captioning has made great progress during recent years, partly due to the optimization of the framework, partly due to the abundance of visual features, and partly due to the introduction of other concepts and structures. In this paper, we propose a novel framework called ITSG, which introduces a visual scene graph to assist the image captioning task. The scene graph is implemented based on Transformer. Compared with the previous GNN-LSTM framework, the Transformer-based visual scene graph alleviates the encoding difficulty. The entire model is homogeneous, the model training setting is easier, and the performance of Transformer is more powerful. In order to reasonably use the Transformer-based scene graph mentioned above, we propose a novel module called Progressive Encoding Module, which inputs the object embedding, attribute embedding and relationship embedding of the scene graph into the encoder, and interacts with the image features in turn to obtain a semantically enhanced visual representation. We conduct experiments and comparisons on the proposed method, proving that our method is efficient.