Image captioning generation based on grid-fusion features

Image captioning is a challenging task that involves generating natural language descriptions for images. In this article the proposed novel approach for image captioning that leverages grid-based fusion features using convolutional networks (CNNs) and Contrastive Language-Image Pre-training (CLIP)-based models. Our method addresses the limitations of existing techniques by combining the strengths of CNNs for capturing spatial relationships in images and CLIP-based models for learning multimodal representations. First, we extract grid-based features from images using CNN, which allow us to capture fine-grained spatial information. These features are then fused with CLIP-based representations, which encode both image and text modalities in a shared embedding space. This fusion process enhances the semantic understanding of the images, leading to more descriptive captions. We evaluate our approach on the MS COCO dataset and achieve state-of-the-art performance in terms of BLEU scores and CIDEr scores. Our method outperforms existing techniques, demonstrating the effectiveness of combining CNNs and CLIP-based models for image captioning.