RoPE-T: Rotary Position Embedding for Enhanced Temporal Modeling in Video-Based Human Action Recognition

Human action recognition requires modeling both spatial information and temporal relationships across video frames. While Transformer encoders are effective for capturing long-range dependencies, standard self-attention mechanisms do not inherently encode the sequential order of input tokens. To address this limitation, we introduce RoPE-T, a RoPE-enhanced temporal Transformer framework for video-based human action recognition. Our approach employs a pretrained EfficientNetV2B0 as a lightweight spatial feature extractor, followed by a temporal Transformer encoder. We apply Rotary Position Embedding (RoPE) directly to the query and key projections within the self-attention module, allowing relative temporal position information to influence the attention scores without modifying the spatial backbone. We also investigate the effect of the rotary base parameter on action recognition performance, showing that its impact varies across datasets and Transformer configurations. Evaluations on the UCF101 and HMDB51 datasets show improvements over our baseline model under identical architectural and training settings, increasing accuracy from 91.51% to 91.59% on UCF101 and from 74.51% to 75.10% on HMDB51. After tuning, the proposed RoPE-T model achieves accuracies of 91.91% on UCF101 and 78.82% on HMDB51, outperforming several previously published action recognition methods on both benchmarks. These results indicate that RoPE is a viable alternative for incorporating relative temporal positional information into CNN–Transformer models for human action recognition.