SIMCAP: SIMilarity-based image CAPtioning

Image captioning is the task of generating descriptive text for a given image, and there are two primary approchaes: fully supervised learning and text-only training. While the fully supervised approach sufferes from dataset limitations and generalization issues, the text-only training approach can address these challenges but faces modality gap between images and text in VLMs. In this study, we propose SIMCAP, an image captioning method that mitigates these modality differences whitin the text-only training. SIMCAP employs similar text retrieval to incorporate text modality and utilizes a meta-network to reflect image modality. Additionally, a mapping network is used to generate prompts that enhance the understanding of the relationship between images and text, facilitating improved text generation. Our experiments show that SIMCAP outperforms existing methods and a through analysis of its componets confirms.