Similarity-Guided Denoising Reconstruction for Unsupervised Image Captioning

Image captioning aims to generate natural and accurate textual descriptions of given images. Although significant progress has been made in image captioning models in recent years, most existing approaches heavily rely on high-quality image-text paired datasets that require expensive human annotation, thus limiting model scalability. Current unsupervised image captioning methods primarily focus on leveraging zero-shot learning capabilities of large pre-trained models (e.g. CLIP, GPT-2), yet still face persistent challenges including modality gaps, inefficient inference, and excessive noise incorporation, which constrain model accuracy and generalization capabilities. To address these limitations, we propose SGDR-Cap ( Similarity- Guided Denoising Reconstruction for Captioning), a novel unsupervised image captioning method that bridges the vision-language modality gap through a similarity-guided denoising reconstruction module. Our method leverages similarity information to guide the reconstruction of authentic text features during caption generation while simultaneously forcing the model to learn how to extract crucial image-relevant features and filter out unnecessary noise information. This enhances both coarse- and fine-grained cross-modal alignment. Furthermore, our approach jointly optimizes denoising reconstruction loss and language modeling loss, ensuring accuracy and fluency, and promoting greater diversity. Extensive evaluations on the MSCOCO and Flickr30K benchmarks demonstrate that our method achieves state-of-the-art results across all major metrics, with the most notable gain on the CIDEr score, improving from 101.1 to 104.4.