VideoMamba-LLaVA: Efficient Video Question Answering with Selective Scanning and Lightweight Inference

Video Question Answering (VideoQA) is a challenging task that combines computer vision and natural language processing to answer questions about video content. While Large Multimodal Models (LMMs) have shown great promise in this domain, their reliance on fixed-size visual tokens extracted from video frames leads to high computational costs. To address this, we propose VideoMamba-LLaVA, a novel framework that integrates the efficient linear complexity model VideoMamba as the visual encoder with a large language model (LLM). VideoMamba employs a selective scanning mechanism to retain and aggregate essential temporal and spatial information, enabling lightweight inference by significantly reducing the input token size to the LLM. Our framework achieves competitive accuracy while maintaining computational efficiency, even with limited tokens. Extensive experiments on VideoQA benchmarks (MSVD-QA, MSRVTT-QA, and TGIF-QA) demonstrate that VideoMamba-LLaVA outperforms state-of-the-art methods with substantial efficiency gains. This work introduces the first application of VideoMamba to VideoQA, establishing a new benchmark for efficient video reasoning.