Bilingual Image Captioning in English and Malayalam Using Convolutional Neural Networks and Large Language Model

This paper introduces a bilingual image captioning system capable of producing meaningful captions in both English and Malayalam. The proposed framework combines a ResNet-18 Convolutional Neural Network (CNN) to extract visual features with a Transformer-based decoder inspired by Large Language Models (LLMs) to generate captions. A notable feature is the incorporation of a bilingual tokenizer, which dynamically selects the output language during inference through a language control flag. The model is trained on the Malayalam Visual Genome 1.0 dataset, which provides aligned image-caption pairs in both languages. Preprocessing steps include image normalization, textual data cleaning, and separate tokenization strategies for English and Malayalam. The decoder generates captions autoregressively, ensuring consistency between training and inference stages. Evaluation is carried out using BLEU, METEOR, and ROUGEL metrics to assess the relevance and fluency of the generated captions. The system achieved BLEU, METEOR, and ROUGEL scores of 0.0456, 0.2365, and 0.2317 for English and 0.0233, 0.2031, and 0.2883 for Malayalam, respectively. The results indicate that the model can effectively handle bilingual captioning, even in low-resource language scenarios. These findings underscore the potential of LLM-inspired decoding strategies for enhancing multimodal understanding across linguistically diverse environments.