Entity-Enhanced Question Representation for Knowledge-Based Visual Question Answering

A good knowledge-based visual question answering (KB-VQA) model requires detailed visual information, semantically clear questions, and relevant external knowledge to address open visual question answering tasks. Although the significant achievements have been made in current KB-VQA works, there are the following challenges: 1) The presence of entities with unclear referents in the question, leading to ambiguity or unclear descriptions. 2) For generative large language models (LLM), the extraction of information from images is insufficient and lacks critical, targeted details. To overcome these challenges, an entity-enhanced question representation for knowledge-based visual question answering model is proposed. Specifically, 1) We employ natural language processing technique to extract entities from the original question and generate related questions focused on these entities. These questions are answered using information from the corresponding image, and the resulting entity-related question-answer pairs are then utilized to enrich the semantics of the original question. 2) We utilize visual language model to enhance image semantics by detecting labels in images based on the MSCOCO dataset label set, while extracting question-guided image descriptions to better represent image semantics. Finally, we apply in-context learning method to generate the final answer using the frozen LLM. Extensive experiments are performed on the OKVQA dataset and achieve superior performance of 61.59% accuracy.