CAVI: A Contrastive Image Captioning Framework with the Contributions of All Visual Information

Image captioning is a multimodal task combining computer vision (CV) and natural language processing (NLP). Contrastive language image pre-training has made significant progress by providing effective multimodal representation. In this paper, we propose a contrastive learning-based perceptual image captioning model that utilizes all visual information.we make improvements in visual encoder, mainly reflected in first storing the visual information encoded by each layer of encoder, and obtaining aligned visual input by averaging to avoid the loss of key visual information.The results show that CAVI improves the ability to describe the relationship between image objects and enhances the quality of generated descriptions. It performs well on the relevant indicators of the Flickr dataset, and also demonstrates certain cross-domain description capabilities on the COCO dataset as a validation set.