Multimodal Transformer for Indonesian Image Text Retrieval

Vision-language model massively trained on image-text pairs. The model can be utilized for image-text retrieval. It retrieves most similar image given a text query, and vice versa. The model trained on abundance of English image captions. Most research pretrained multimodal encoder for English caption dataset but limited to other language. This research aimed to train Transformer-based model to large Indonesian image-text pairs for image-text retrieval. We trained to large dataset, so it can further be utilized for downstream tasks or transfer the learned parameter to a smaller dataset. Dataset were acquired from translated COCO and Flickr dataset. The translation was done by Google Translate platform as in NusaCrowd image captioning catalogue. The employed pretrained models in this research were CLIP, ALIGN, and BLIP as the top multimodal model benchmark. We also constructed CLIP-like architecture utilizing ViT, DeIT, and BeIT as image encoder with IndoBERT and ModernBERT as text encoder. Images were resized and augmented randomly. Texts were cleaned from punctuation and lowercased. Models performed well and achieved good retrieval performance on the test set. The highest R@10 on COCO test set was by BeIT-IndoBERT model with 72.20% R@10 for image-to-text and 65.21% R@10 for text-to-image retrieval. This research can be improved by employing on more refined dataset with human-annotated image captions by Indonesian annotators. Dataset scraped from Indonesian online documents can also be leveraged in future research. These models can be further trained on downstream tasks, including image captioning and visual question answering.