Visual Question Answering: Convolutional Vision Transformers with Image-Guided Knowledge and Stacked Attention

Visual Question Answering (VQA) have shown promise in understanding and answering questions about images, but they struggle with better image representation techniques and complex reasoning tasks that require external knowledge. This work addresses the mentioned limitations by incorporating an image-guided knowledge attention mechanism for improved reasoning capabilities. The objective is to enhance VQA models by integrating external knowledge and to develop a robust approach that takes advantage of hybrid transformers for image representation. To achieve these goals, a VQA model CvT-KB is proposed, utilizing a Convolutional Visual Transformer (CvT) for rich visual feature extraction, a stacked attention network for effective multi-modal fusion of image and text information, and an image-guided knowledge attention mechanism to incorporate relevant knowledge from ConceptNet based on entities identified in the given question. The model demonstrates improved accuracy of 52.2% and WUPS score of 52.81% on the OK-VQA dataset compared to baseline models, and highlight the ability to utilize knowledge and achieve a deeper understanding of visual scenes.