Synthetic View Augmentation for Sign Language Recognition

Sign language, as a visual language, is inherently spatial and dynamic, characterized by various hand shapes, movements, facial expressions, and body postures.In real-world scenarios, sign language recognition (SLR) systems are deployed in diverse environments with varying lighting conditions, backgrounds, and user positions, and models trained on data from a single viewpoint often struggle to generalize effectively to new perspectives.To address these limitations, this paper focuses on cross-view SLR and proposes a synthetic view augmentation pipeline to facilitate SLR models learning viewpoint-invariant features during training, thereby improving their performance on unseen viewpoints.Additionally, we introduce the XV-SLR model, which incorporates multi-modality information further to enhance the robustness of cross-view sign language recognition.The proposed method achieves a top-1 accuracy of 56.87% under the RGB setting and 57.97% under the RGB-D setting and outperforms other approaches in the Cross-View Isolated Sign Language Recognition Challenge (CV-ISLR).Code is available at https://github.com/VIPL-SLP/XV-SLR.