Word Level LSTM CNN based sign language to sentence translator

In this study, a robust and lightweight model for sign language gesture recognition is presented, compatible with various sign language standards. This approach addresses challenges such as tracing, occlusion, and computational efficiency by leveraging the MediaPipe framework to extract and preprocess features from the left hand, right hand, face, and body, converting them into NumPy arrays. A layered Long Short-Term Memory (LSTM) network is employed to combine with a Dense Network of Neurons which aid from the heart of Convolution (CNN) layer help and aid the model learn for language-agnostic gesture detection. This methodology achieves high accuracy, even with a limited dataset, demonstrating the model’s efficiency and effectiveness. The model’s simplicity results in a smaller set of trainable hyperparameters, leading to faster training and improved real-time performance. Additionally, the model incorporates a key-to-text pipeline using the T5 module, generating coherent sentences from detected gesture sequences for real-time translation. This enables real-time detection and translation of sign language, applicable across various domains. The model exhibits 95% classification accuracy on a test dataset of organically generated set of datasets and a loss percentage of 0.3% on the same set of classification. Further the sentence formation utilizing a transformer architecture of T5, enables coherent sentence formation from these keywords.