Multi-Branch Information Network for Advanced Image Captioning

Understanding images from different perspectives and performing advanced semantic consistency transformation across modalities are key elements in generating natural language descriptions that are relevant and coherent with the current image captions. Insufficient understanding or errors in interpreting visual features of images can lead to inconsistent descriptions or generate hallucination problems. Moreover, current techniques often employ attention mechanisms to achieve semantic alignment in different representation spaces while overlooking the underlying heterogeneity of multimodal data. In this paper, we propose a Multi-Branch Information Network for Advanced Image Captioning that integrates comprehensive visual semantic understanding and advanced semantic consistency transformation into a single image captioning model architecture. Technically, we first utilize a cross-modal retrieval model to obtain fine-grained contextual descriptions for each image region, aiming to compensate for and alleviate hallucination. Next, we introduce a novel mapping network that reconstructs visual information in the textual space and achieves consistency transformation between visual and textual modalities within the textual domain. Empirical evidences show that, ours outperforms the latest approaches on the COCO dataset and achieves the best CIDEr score of 138.4% in the Karpathy test set.