LLMs Encounter Critical Elements Prompts: Semantically Guided Partial Supervision Skeleton-Based Action Recognition

Semantically guided skeleton-based action recognition is a crucial but challenging task in the data processing with RGB and depth sensors. The generation of action descriptions is facilitated by the pretrained language model, thereby imbuing the skeleton with essential contextual semantics. However, it also presents two challenges that require careful consideration: (1) efficiently and automatically generating the core semantics of actions, and (2) extending the applicability of supervised learning when only partial labels are available. Therefore, we propose Critical Elements Prompts for Partial Supervision in Skeleton-based Action Contrastive Learning (CEPCLR) as a significant attempt to break through the existing limitations of skeleton and semantic multimodal fusion. Firstly, we design action-specific prompts that enable large language models (LLMs) to generate general semantic descriptions both efficiently and accurately, followed by iterative prompting that mines discriminative descriptions through controlled, rule-based emergence. Secondly, we adopt Language-Skeleton Contrastive Learning for multimodal fusion, which significantly enhances the integration of positive and negative action pairs. Notably, we introduce a novel partially supervised learning task for skeleton data, where labels are used selectively for semantic generation but are not involved in the classification tasks. The proposed CEPCLR achieves strong results on standard datasets like NTU RGB+D, NTU RGB+D 120, and PKU-MMD, effectively addressing missing contextual information in human skeleton and action expressions.