Long- and short-term spatio-temporal transformer for skeleton-based action recognition

Existing approaches for skeleton-based action recognition often focus on either short- or long-term dependencies, whereas the integration of both aspects remains underexplored. In addition, insufficient attention has been paid to extracting correlations among different joint points across adjacent frames in many methods. To address these limitations, we propose a long- and short-term spatio-temporal transformer (LS-STFormer), featuring three key components: entity rearrangement (ER), local temporal encoding (LTE), and long- and short-term spatio-temporal self-attention (LSSA). The ER module enhances generalization by shuffling entity orders while preserving the semantic consistency of unordered interactive subjects. The LTE module reduces local feature redundancy through a decoupled learning strategy and explicitly models inter-joint correlations across consecutive frames via hybrid positional encoding. The LSSA module incorporates a dual-path design and shift operations to jointly capture global long-term dependencies and local short-term spatiotemporal patterns, thereby improving recognition accuracy. Extensive experiments on NTU RGB+D and NTU RGB+D 120 datasets demonstrate that LS-STFormer achieves state-of-the-art performance, outperforming existing methods by significant margins.