Efficient image captioning with CLIP and GPT-2 via prefix mapping

Image captioning, which generates natural language descriptions for visual content, represents a convergence of computer vision and natural language processing. It is utilized in assistive technologies and automated content generation. This paper investigates an efficient image captioning framework that integrates a frozen Contrastive Language-Image Pretraining (CLIP) visual encoder with a Generative Pre-trained Transformer 2 (GPT-2) language decoder through a lightweight prefix mapping network. By leveraging CLIP’s robust visual embeddings and a semi-decoupled architecture, this approach reduces training costs while ensuring semantic alignment. We investigate two mapping strategies: Linear MLP and Transformer, to transform CLIP embeddings into prefix tokens, conditioning GPT-2 for autoregressive caption generation. This approach, evaluated on the Microsoft COCO Captions and Conceptual Captions datasets, achieves competitive performance, with the Transformer variant achieving 33.53 BLEU-4, 113.08 CIDEr, and 21.05 SPICE on COCO, outperforming several baselines. Training times are significantly diminished to 6–7 hours, compared to 32–1040 hours for baselines, illustrating superior efficiency. An ablation study on prefix length optimizes performance by balancing expressiveness and scalability. This work offers a scalable solution for effective image captioning and contributes to the advancement of vision-language systems.