An Empirical Study on Utilizing Large Language Models for Bengali Image Caption Generation

An exemplary caption not only describes what is happening in a particular image but also denotes intricate traditional objects in the image by their local representative terms through which the native speakers can recognize the object in question. A caption that fails to accomplish the latter is not effective in conveying proper utility. To ensure caption locality, we aim to explore the potential of Large Language Models (LLMs) in Bengali image captioning, which have lately shown promising results in English language caption generation. As a first for the Bengali language, we utilized CLIP (Contrastive Language-Image Pre-training) encodings as a prefix to the captions by employing a mapping network, followed by fine-tuning BanglaGPT, a Bengali pre-trained large language model to generate the image captions. Furthermore, we explored vision transformer-based encoders (ViT, Swin) with BanglaGPT as the decoder. The best BanglaGPT-based model outperformed the current benchmark results, with BLEU-4, METEOR, and CIDEr scores of 54.3, 39.2, and 95.9 on the BanglaLekha dataset and 67.4, 36.6, and 76.9 on the BNature dataset.