Advanced Video Classification for Sign Language Translation Using CNN-LSTM

Sign Language (SL) is a highly effective means of communication for the hearing and speech impaired population. Automated SL Translation (SLT) is proposed to bridge the gap in communication by converting gestures to words or text. Although most of the prior work has targeted individual signs, this work focuses on the challenging task of continuous SLT from video-level sentence information. This paper suggests a deep learning architecture that learns spatio-temporal features from SL videos using MobileNetV2 for spatial representation and a Bidirectional LSTM for temporal modeling. The model is trained on a continuous Indian SL (ISL) dataset of single- and double-handed dynamic gestures with 87.35 % training accuracy and 83.64 % validation accuracy. Besides that, an interactive web app employing Streamlit is produced that accepts input video and provides outputs as corresponding SL sentences in realtime. Unlike previous word-level only models, our model works with actual world, sentence-level sequences, thereby improving context-aware translation.