Pose-Guided Focal Loss for Enhancing Vision Transformers in Continuous Sign Language Recognition

Continuous Sign Language Recognition (CSLR) aims to convert sign language videos into a sequence of gloss annotations. A standard Connectionist Temporal Classification (CTC) based CSLR model comprises two primary modules: a visual encoder and a sequential decoder. This research evaluates the effectiveness of current deep learning architectures in CSLR by systematically replacing the visual encoder with CNN, 3D-CNN, and Vision Transformers (ViT), while also exploring different sequential decoders based on RNN and Transformer architectures. Findings indicate that a CNN+Transformer structure achieves the highest accuracy, whereas ViT-based models underperoms due to insufficient training data and noisy supervision. To mitigate this, the authors introduce Pose Guided Focal Loss (PGFLoss), a novel loss function that leverages pose heatmaps to refine the self-attention layers in ViT. Experiments demonstrate that PGFLoss substantially enhances ViT performance on Word Error Rate (WER), from 24.1 % to 21.1 % in PHOENIX14 dataset, closing the performance gap with leading methods. The code is available here.