Adaptive and Guided Attention in Transformer for Visual Question Answering

Transformer-based techniques for Visual Question Answering (VQA) have shown commendable prowess in modeling global relationships. However, they often encounter a decrement in accuracy due to their limitations in capturing local dependencies. In response to this challenge, we propose a novel solution, the Adaptive and Guided Attention in Transformer (AGAT), tailored for the VQA task. AGAT constructs an adaptive selfattention method, where a neighborhood prediction module is adopted to dynamically fine-tune attention span and weight by sampling the query token from a default neighborhood. This innovative approach enables the comprehensive modeling of long-term dependency relationships and facilitates extracting contextual information from diverse image regions. The guided attention, which leverages information from the question encoder, is further integrated. It interplays between different modalities multiple times to align the text and image. Experimental results demonstrate that AGAT outperforms the representative methods significantly in terms of accuracy.