Performance Analysis of Vision-Language Pre-Training Models on Unbiased VQA Tasks

This study evaluates the performance of Visual Language Pre-training (VLP) models on a colour recognition task using unbiased Visual Question Answering (VQA) datasets. We focused on three state-of-the-art multimodal VLP models: Vision-and-Language Transformer (ViLT), Generative Image-to-text Transformer (GIT), and Bootstrapping Language-Image Pre-training (BLIP). To assess these models, we developed three novel balanced datasets with binary question-answer pairs, each designed to detect the colour of objects. The datasets differ in image creation methods, including fully synthetic images, photographs of coloured objects, and varied objects with realistic backgrounds. Our results indicate that dataset characteristics significantly influence model performance. ViLT consistently performed well on simpler binary datasets, achieving an accuracy of around 62%, but struggled with more complex formats. GIT showed stable performance in simpler binary datasets but had difficulty with complex questions, especially in combined colour identification tasks. BLIP is highly adaptable, excelling in complex question formats with perfect accuracy in colour identification tasks. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>