Arabic Sign Language Recognition for Differently-Abled Individuals Using Vision Transformers and Temporal Analysis

This paper introduces an improved Arabic Sign Language Recognition (ArSLR) system that aims to facilitate increased accessibility to communication for the deaf and hard-of-hearing population. The system utilizes a Shifted Window Transformer for spatial feature extraction, which is a cutting-edge architecture that uses a computationally efficient hierarchical and shifted window mechanism to extract both local and global dependencies from sign language gestures. With Long Short-Term Memory (LSTM) networks for temporal modeling, the system attains state-of-the-art performance in Arabic alphabet and digit recognition. A two-stage methodology approach is utilized, with the Transformer layers first frozen to retain general visual representations and then fine-tuned to learn domain-specific sign language characteristics. This strategy effectively balances between retaining trained features and adaptation to the target domain, providing better generalizability to different signers. The suggested system is benchmarked on the KArSL dataset containing 502 signs carried out by three expert signers. Intensive data augmentation practices are utilized in order to emulate real-world scenario conditions like poor brightness, movement blur, as well as jittering over time, which contributes to the enhancing robustness of the system. The outcomes illustrate high levels of near-perfect accuracy in signer-dependent situations, with average accuracies of 99.56% for Arabic alphabets and 99.56% for digits. In signer-independent conditions, the system achieves 72.53% and 69.55% average accuracies for alphabets and digits, respectively, better than methods like skeletal feature transformation techniques.The results demonstrate the promise of the proposed system in being a viable, scalable option for real-world application, empowering inclusivity and accessibility for individuals with disabilities.