Toward Real-Time Recognition of Continuous Indian Sign Language: A Multi-Modal Approach Using RGB and Pose

Sign language recognition (SLR) involves translating visual gestures into meaningful text or speech, bridging the communication gap between signers and non-signers. However, real-time recognition remains a critical challenge due to variability in signing speed, subtle hand gestures, and the computational complexity of processing video data in real time. Existing methods often rely on encoding RGB videos into latent representations, making them unsuitable for real-time interaction. To address these challenges, we present SignFlow, a network for real-time recognition of continuous sign gestures. Our approach introduces a novel pre-processing technique to down-sample video streams, ensuring compatibility with varying frame rates across different devices. For the first time, the core components of our network are pre-trained on domain-specific Indian Sign Language (ISL) data. The CNN is pre-trained using ISL word videos, while the Transformer is pre-trained on Mediapipe pose estimates from ISL videos. This pretraining effectively captures the nuances of hand shapes and body movements unique to ISL, significantly enhancing sentence recognition. SignFlow combines a pre-trained CNN for feature extraction and a Transformer for learning the temporal dynamics of continuous signing. The framework is fine-tuned end-to-end using Connectionist Temporal Classification. For evaluating the real-time models, we have introduced a detection rate metric, which measures how accurately individual gestured words are recognized within a sequence, regardless of their order. SignFlow achieves a Word Error Rate (WER) of 19 on the Continuous ISL dataset, demonstrating its effectiveness for real-time ISL recognition. Additionally, it shows competitive performance on the German Phoenix 2014 and Phoenix 2014T datasets.