Action-JointsLLM: A Novel Transformer-Based Framework for Efficient Fine-Grained Action Detection

Fine-grained action detection remains a challenging and actively studied problem. While previous methods predominantly rely on convolutional neural networks (CNNs), their limited modeling capacity and high computational cost restrict their effectiveness in fine-grained settings. To overcome these challenges, we propose Action-JointsLLM, a novel framework that leverages joints extracted from visual data and large language models (LLMs) for accurate and efficient fine-grained action detection. Specifically, we introduce a Joint2Text module that transforms sequences of human joints into natural language descriptions, enabling direct compatibility with LLMs. Additionally, we design an attn-pooling mechanism to identify the contributions of each joint and time step across the motion sequence, enabling more accurate action analysis. Our approach demonstrates strong generalization across different LLM backbones and achieves state-of-the-art performance on the PD-WALK dataset, using Parkinson’s disease detection as a representative fine-grained action recognition task.