Continuous Sign Language Recognition and Translation Using Deep Learning Approach

This paper introduces a novel approach for sign language gesture recognition which leverages the fusion of spatial and skeletal features. The proposed system uses spatial feature extracted from RGB images of the signer via ResNet and skeletal features are obtained via MediaPipe, thus achieving both precise and complete data representation. A robust preprocessing pipeline is adopted for processing input data. It includes data cleaning, normalization, denoising, and frame sequence alignment. The hybrid TCN-LSTM model is the core of the system. The concatenated feature vector with both spatial and skeletal information is fed into the model. The system undergoes intensive training and testing to guarantee excellent accuracy and performance in the prediction of semantic meanings of gestures. The TCN captures local temporal dependencies and LSTM handles long range dependencies. Results demonstrate a considerable gain in recognition accuracy, which proves the effectiveness of this strategy. This research makes a contribution toward real-time gesture recognition systems, with applications relevant to the facilitation of communication for individuals with hearing or speech impairments.