CSCCap: Plugging Sparse Coding in Zero-Shot Image Captioning for Complex Background Images

Recently, zero-shot image captioning based on pre-trained vision-language models (VLMs) and large language models (LLMs) has made significant progress. However, experiments and experience have demonstrated that these methods do not perform well when dealing with complex images, as the analysis of the image subject can be affected by intricate backgrounds. To address this issue, we constructed a learnable convolutional sparse coding (CSC) model to learn sparse representations of images. By devising a novel initialization strategy to optimize the iterative structure of the CSC model, we constructed an improved CSC model named RepCSC. By incorporating RepCSC, it is possible to extract the sparse representation of the image, effectively removing unnecessary background features and highlighting the main subject of the image, thus enhancing the effectiveness of image-to-text generation. Numerous experiments have shown that models incorporating sparse encoding perform competitively in zero-shot image captioning over other methods.