Adapting Multimodal Large Language Models for Video Question Answering by Capturing Question-Critical and Coherent Moments

Multimodal Large Language Models (MLLMs) have demonstrated remarkable abilities in image-language reasoning. However, they deal with Video Question Answering (VideoQA) insufficiently, especially for questions demanding causal-temporal reasoning. Typically, they directly concatenate features of uniformly sampled frames as visual inputs for VideoQA. This gives rise to two challenges. For one thing, uniformly sampled frames are discrete and separately distributed across different timestamps, disrupting the coherence of question-critical events or actions. For another, it considers every scene within videos equally and introduces redundant frames that may distract the model from discovering the truth. Towards this, we highlight the importance of identifying continuous frames that are crucial for answering the questions, and propose a lightweight and differentiable <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Co</i>herence <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Re</i>cognizer (<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CoRe</i>) to achieve this. Guided by the semantics of questions, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CoRe</i> computes scores recording the relevance between each frame and the question, and selects a set of continuous frames with the highest scores for answer prediction. Additionally, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CoRe</i> encodes the unselected frames into a short and coarse-grained representation as a completion of the general context. Equipped with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CoRe</i>, we can efficiently fine-tune the current MLLMs for VideoQA in an end-to-end manner, without suffering from the problems of incoherence or distraction. Extensive experiments demonstrate that our method achieves substantial improvements on several VideoQA benchmarks.