MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answering
Visual Question Answering (VQA) requires reasoning across visual and textual modalities, yet Large Vision-Language Models (LVLMs) often lack integrated commonsense knowledge, limiting their robustness in real-world scenarios.To address this, we introduce MAGIC-VQA, a novel framework that enhances VQA by systematically integrating commonsense knowledge with LVLMs.MAGIC-VQA employs a three-stage process: (1) Explicit Knowledge Integration from external sources, (2) By-Type Post-Processing for contextual refinement, and (3) Implicit Knowledge Augmentation using a Graph Neural Network (GNN) for structured reasoning.While GNNs bring greater depth to structured inference, they enable superior relational inference beyond LVLMs.MAGIC-VQA bridges a key gap by unifying commonsensse knowledge with LVLMdriven reasoning, eliminating the need for extensive pre-training or complex prompt tuning.Our framework achieves state-of-the-art performance on benchmark datasets, significantly improving commonsense reasoning in VQA.Our implementation is open-sourced on GitHub
Authors
- Shuo Yang (ORCID: https://orcid.org/0000-0003-0190-3319)
- Eduard Hovy (ORCID: https://orcid.org/0000-0002-3270-7903)
- Siwen Luo (ORCID: https://orcid.org/0000-0003-0480-1991)
- Cheon Goo Han
Publication Details
- Published
- 2025-01-01
- DOI
- https://doi.org/10.18653/v1/2025.findings-acl.872
- Citations
- 1
- Primary Topic
- Multimodal Machine Learning Applications
- Type
- article
- Field-Weighted Citation Impact
- 1.28