VEGCAP: A Visual Entity-Guided Image Caption Generation method for Visual Question Answer

The task of Visual Question Answering (VQA), particularly in datasets like OK-VQA, demands the integration of external knowledge to provide accurate answers. In recent years, approaches that leverage large language models as implicit knowledge bases have demonstrated remarkable progress. However, a significant challenge for these approaches is effectively bridging the gap between visual data and language models. Representing image content solely through image captions and object labels may overlook crucial information and introduce noise, failing to adequately capture the complexity and details of the image, thereby affecting the accuracy of the model’s responses. To address this issue, this paper proposes VEGCAP, a visual entity-guided image captioning method. By analyzing natural language questions, VEGCAP accurately identifies the visual entities in the image that need to be emphasized, generating more targeted image descriptions.Compared to general image captioning methods, VEGCAP effectively minimizes interference from irrelevant information and highlights key details related to the question. Experimental results demonstrate that the proposed method achieves state-of-the-art accuracy on knowledge-based VQA datasets, OK-VQA and A-OKVQA, reaching 60.9% and 59.2% on the validation sets, respectively.