Bool Prompt with Decomposition and Enhancement: Zero-Shot VQA Based on PVLMs

Zero-Shot Visual Question Answering (ZSVQA) aims to answer questions about images without prior training on explicit image question pairs. Most existing methods usually apply Pre-trained Visual and Language Models (PVLMs) by designing prompts to convert questions into predefined input templates, which (1) ignores the text details and associations of the question when the question is complex, hindering comprehensive understanding, and (2) does not pay attention to local information of image, resulting in overlooking some details of the image that are important to the question when the image content is particularly complex or requires detailed observation. To address these challenges, we propose the Bool Prompt with Decomposition and Enhancement (BPDE) framework for ZSVQA. Specifically, we propose the Bool Sub-Questions Generating module to extract keywords from the original question and generate captions from the image, then use these keywords and captions to guide the transformation of original questions into simpler bool sub-questions, which focus on a specific logical point or piece of information, and guided from the captions can provide the model with local visual information, thereby enhancing the model’s understanding of complex questions and attention to local visual information. Additionally, an Adaptive Sub-Questions Selecting mechanism is designed to ensure non-redundant selection and that the meanings of the sub-questions can cover the original question. Extensive experiments on VQAv2 and AOKVQA demonstrate that the proposed approach performs favorably against the state-of-the-art methods.