Unifying Vision-Language Models and Knowledge Graphs for Zero-Shot Counter-Intuitive Reasoning in Images

Counter-intuitive visual reasoning aims to recognize and interpret why weird, unusual, and uncanny image content violates common sense or lacks a basis in reality. It is intuitively appealing to employ pre-trained Vision-Language Models (VLMs) due to their impressive zero-shot capability for understanding and reasoning. However, the inherent gap between counter-intuitive images and pre-training images hinders VLM from perceiving commonsense-violating entity relationships that are not seen during pre-training. To address this challenge, we propose a framework that unifies VLM and knowledge graphs together to explicitly incorporate commonsense knowledge into counter-intuitive reasoning. Starting with extracting fine-grained visual entities and their relationships using a VLM, we then reason about commonsense knowledge of the entities and relationships through knowledge graph completion, and finally interpret the deviations between the visual content and the commonsense knowledge using a pre-trained Large Language Model (LLM). Experiments on the visual dataset WHOOPS! demonstrate the effectiveness of our method.