LTD-NET: Lightweight Two-Stream Decoder Net for Action Recognition

Pretrained vision-language models have exhibited substantial efficacy in video action recognition. Nevertheless, the spatiotemporal redundancy inherent in video, coupled with the spatial and temporal inconsistency of visual cues pose significant challenges for practical applications. To address this issue, we introduce a lightweight two-stream decoder network (LTD-Net) that unifies the spatiotemporal features within each decoder layer and effectively reduces redundancy across both temporal and spatial dimensions. Specifically, in the spatial stream, we design a novel temporal token reduction module (TTR module). This module utilizes temporal attention to select key tokens from the same spatial positions across multiple frames, aggregating them into single-frame tokens to reduce temporal redundancy. In the temporal stream, we propose a novel spatial token reduction module (STR module). This module employs spatial attention to choose essential spatial points’ information within frame, reducing spatial redundancy. By reducing spatiotemporal redundancy, the computational complexity of our LTD-Net is less than 3% of the pretrained model. Experiments on Kinetics-400, UCF-101, and HMDB-51 demonstrate that our LTD-Net achieves competitive results comparable to the state-of-the-art methods with lower training costs.