STA+: Spatiotemporal Adaptation with Adaptive Model Selection for Video Action Recognition

Recent breakthroughs in video models have achieved remarkable success by integrating vision transformers into the video domain through adaptation. However, prevalent approaches in existing literature often entail a single branch, incorporating additional spatiotemporal learning modules. This approach cannot fully satisfy the unique and targeted learning requirements of temporal and spatial representations. It lacks a well-balanced spatiotemporal understanding of videos, particularly in recognizing human actions, which requires a comprehensive understanding of space and time. In this study, we introduce a novel two-stream architecture named SpatioTemporal Adaptation with Adaptive Model Selection (STA+) to achieve synchronized spatiotemporal understanding of video with only RGB inputs. We propose an adaptive model selection method based on the intrinsic categorization judgment of the samples to enhance overall performance. We validate the effectiveness of our proposed method through rigorous experiments conducted on public benchmarks such as Diving-48 and Something-Something- V2. Our method consistently achieves competitive performance on these datasets.