Improving VQA Counting Accuracy for Post-Flood Damage Assessment

Flood disasters cause significant damage and loss of life globally, demanding efficient disaster response mechanisms. Visual Question Answering (VQA) systems, which integrate visual and textual data to answer user queries, offer a promising solution for flood disaster management. Previous deep learning methodologies showcased limitations in answering counting related questions. To address this, we propose DiViNeCaps (DistilBERT Vision Network with CapsuleNet classifier), a novel VQA network leveraging DistilBERT as the language model and Vision Mamba for image embedding. Our approach features an innovative attention-based fusion module, combining self-attention, gated attention, and cross-attention, followed by a Capsule network classifier. Our network improves counting accuracy and overall performance compared to state-of-the-art methods. Experimental results demonstrate that our architecture improves accuracy by 2.95% for Simple Counting and 3.38% for Complex Counting questions on the FloodNet dataset.