Retention Enhanced Cross-modal Attention for Multi-Hop VQA

Exploring multimodal information from external knowledge bases in Visual Question Answering (VQA) reasoning tasks presents a significant challenge. Current methods, such as attention-based and graph-based approaches, have limitations in effectively capturing contextual key information. It is essential to enhance the model’s cognitive understanding of the associations between the question and the corresponding external knowledge to improve accuracy. To overcome these challenges, this paper introduces RECA, a Retention Enhanced Cross-modal Attention method for multi-hop VQA inspired by the hypergraph. Specifically, the contextual information after cross-modal attention fusion are further enhanced by the proposed retention module based on Retentive Network. Finally, we introduced a mimicry loss through model distillation. This enabled our model to enhance collaborative learning and improve generalisation performance. Our method is evaluated on KVQA, FVQA, PQ, and PQL datasets, demonstrating state-of-the-art performance.