Evaluations of various textural models on visual question answering counting task

Multimodal learning is a machine learning field that mainly focuses on connecting features in different modalities, and VQA counting task is a special kind of multimodal task aimed at answering a particular object’s count in a given image according to the query. Hoping to improve the performance of VQA counting models further, we propose three methods to modify existing work. In VQA tasks, textual and image features are processed separately before the alignment process, whereas the structure of our baseline model’s text processor is very simple, using only a GRU layer after text embedding. As a result, when processing textual information, we introduce a pretrained BERT model and the self-attention mechanism to improve the complexity and performance of the model. Apart from that, in VQA tasks, image and textual features that have been extracted are aligned and fused in feature fusion stage, whereas the baseline model lacks trainable parameters when doing fusion. To add more trainable parameters, we apply the fusion method used in MLB to the model. After applying these modifications, we evaluate the performance of our works separately on the training set and validation set of the VQA v2 dataset, whereas the results show that in spite of the fact that we introduce later methods to improve the existing model, our methods do not achieve better performances and thus we analyze the possible causes of each result of our methods in detail. To sum up, the main contribution of our work lies in proposing three methods based on the deficiencies of an existing model to improve performance in VQA counting tasks and giving detailed analyses of our results.