Spatial–Temporal Clue Reasoning Chain for Long Video Question Answering

Existing Video Question Answering (VideoQA) methods face tremendous challenges when dealing with longer videos. On the one hand, long videos contain rich and diverse information at different spatial-temporal scales, requiring the VideoQA model to mine multi-granularity visual clues conditioned on the specific question. On the other hand, the model needs to effectively reason the spatial-temporal relationships of complex events that span large temporal intervals. To address the above challenges, we propose a novel spatial-temporal clue reasoning chain to effectively reason multi-granularity visual clues and predict the answer in complex events. Specifically, we extracts question-related visual clues from video clips, frames and regions in a “step-by-step" manner. Further, we propose an asynchronous clue selector with self-excitation loss to perform more effective selections by (1) facilitating the identification of question-related visual clues and (2) enhancing the diversity of selections. Next, we introduce cross-modal fusion between the question semantics and the selected multi-granularity visual clues to learn the detailed interactions and then predict the answer. Extensive experiments on multiple VideoQA datasets demonstrate the effectiveness of our method.