GestureSpeak: An Efficient Real-time Finger Spelling Recognition Model

In this paper, an efficient real-time finger spelling recognition system which designed for bridging communication gaps for the deaf and hard-of-hearing communities is presented. The traditional sign language recognition systems are more likely focusing on accuracy rather than the speed and efficiency in low specs devices. This leads to limiting the usage of the traditional system as it needs high specs of the users’ devices. To solve this issue, a model balancing both accuracy and response time is proposed. The proposed approach leverages n-gram modeling and Knowledge Distillation techniques to enhance performance and accuracy. The system integrates these methodologies to capture the temporal dependencies between gestures and reduce the computational burden typically associated with deep learning models. Different variants of EfficientNetv2 and MobileNetV3 architectures, pre-trained on the ImageNet dataset, were utilized to achieve a balance between efficiency and accuracy. Experimental results demonstrate that the proposed system maintains high accuracy with significantly reduced computational requirements, making it suitable for practical applications in resource-constrained environments such as mobile devices. This research contributes to the field by introducing a novel combination of n-gram modeling and Knowledge Distillation in the context of finger gestures recognition in sign language, offering a viable solution for real-time applications. Unlike other normal works focusing on the parts of sign language which contains poses of arms and gestures of arthrosis, we concentrate more on the hand gesture-based sign language recognition. Through our work, we believe that we can promote the communication between sign languages with the efficient real-time finger spelling recognition model in a way of hand gesture-based.