Dual-level question-guided image fusion network for visual question answering

The Visual Question Answering (VQA) task involves open-ended questions with varying granularities, and the requirements for image feature representations are inconsistent. To better answer all types of questions, this paper introduces a VQA model based on a two-layer question-guided image fusion network. First, two different image features are used to generate corresponding guiding vectors. Then, an image filtering module is applied to mask the attention maps with high entropy. Finally, the guiding vectors are cross-attended, effectively accommodating questions of different granularities and fully utilizing the relationships between answering information. The DQIFN model leverages the advantages of both features to generate answers, balancing both contextual and object information. Additionally, it enhances the understanding of associations between objects with varying granularities when the question is introduced, addressing the issue of computational explosion. Extensive experimental results show that the proposed method improves the completeness of visual features and enhances the accuracy of answers in VQA tasks. Due to the presence of these two modules, CSAM can focus on high-level semantic features correctly and avoid image noise. Experimental results on the VQA 2.0 datasets demonstrate that the DQIFN model outperforms existing state-of-the-art models.