Retrieval-augmented prompts for text-only image captioning

Supervised image captioning methods have made significant progress, but high-quality human-annotated image-text paired datasets are costly to collect. Recently, pre-trained vision-language models, such as CLIP, have shown exceptional performance in cross-modal associations, offering novel solutions for image captioning, such as zero-shot captioning through purely textual training. However, the modality gap exists between image and text presents an obstacle to cross-modal alignment within text-only captioning. Moreover, insufficient visual understanding and over-reliance on textual data in training lead to hallucinations (e.g., object misidentification and inaccurate object counts), resulting in irrational captioning. To tackle these issues, this paper presents RAPCap, a text-only method with retrieval-augmented prompts for text-only image captioning. Specifically, RAPCap enhances the language model's understanding of images by incorporating similar captions obtained through retrieval augmentation, thereby alleviating hallucinations. During inference, RAPCap translates the image to textual space to bridge the modality gap. Experimental results demonstrate that RAPCap achieves a new state-of-the-art performance on the Flickr30k and performs competitively on the MSCOCO compared to previous zero-shot captioning.