Hand-Aware Masked Graph Convolutional Network for Skeleton-based Sign Language Recognition

Sign language recognition (SLR) presents significant challenges due to the complexities in modeling the intricate interactions between hand gestures and other body movements, as well as the presence of redundant and noisy temporal features. Existing Graph Convolutional Network (GCN)-based methods, while promising in capturing skeletal dynamics, are limited by their reliance on fixed skeletal structures and uniform frame importance, which hinder their effectiveness in addressing these issues. To overcome these challenges, we propose a novel Hand-Aware Masked Graph Convolutional Network (HAM-GCN), which introduces two key innovations: an adaptive dynamic hand graph and an adaptive masking mechanism. The hand graph effectively captures fine-grained interactions between hand gestures and body movements, enhancing the accuracy of sign language representations. The adaptive masking mechanism, driven by hand-specific confidence scores, dynamically adjusts the importance of individual frames, emphasizing informative ones while suppressing noisy or redundant features. We validate HAM-GCN through extensive experiments on three publicly available sign language datasets, achieving state-of-the-art performance and surpassing existing methods. The source code is publicly available at https://github.com/Gavin113/HAM-GCN.