Temporal Text Prompts for Skeleton-based Action Recognition

Skeleton-based action recognition has a wide range of applications in intelligent robotics, such as assistance for the elderly and disabled, human-robot interaction, and surveillance. Inspired by the success of vision-language pre-trained models, some works attempt to employ text prompts of actions as the prior knowledge to improve the performance of models. However, these text prompts focus more on spatial information and ignore the temporal knowledge of actions, while skeleton features contain temporal and spatial information. This not only leads to a semantic gap between visual and textual modalities but also limits the model from learning temporal cues of actions. To address this issue, we propose a Visual-Language Aligned spatial-temporal Feature Enhancement framework (VLA-FE), which applies both temporal and spatial text prompts of human actions to guide the model in learning more discriminative representations. VLA-FE adopts a two-branch multi-modal training strategy, which is decomposed into spatial-aware and temporal-aware processes. Moreover, We apply two contrastive losses correspondingly to guide this process. Experiments show that the proposed method achieves promising performance on the NTU RGB+D and NW-UCLA datasets.