A Novel Explainable Deep Learning Model for Sign Language Recognition and Translation

Sign language is essential for deaf and hard-of-hearing individuals, but real-time access to it needs special tools or technology due to the expertise required for interpretation. This research work proposed a novel deep learning system for Sign Language Recognition and Translation. The proposed model utilizes m-ResNet50 for extracting features and LSTM networks to analyse sequential gestures, thus capturing both spatial & temporal gesture specifications proficiently. This work introduces a 3D Class Activation Map (3D CAM) visualization tool to demonstrate the regions that influence model decision-making when detecting gestures during recognition. The model gains superior gesture recognition performance through this spatio-temporal analysis using CNN-RNN model. The model is trained and validated with Sign Language MNIST dataset that originated from MNIST. The m-ResNet50 model shows more than 95% success in automatic interpretation between various signs which leads to real-time speech or textual outputs across benchmarked dataset. Experimental findings reveal that the model maintains its strong performance in terms of test accuracy, achieving 98.01% while reaching a test loss of 0.0690 along with a validation accuracy of 97.99% that reaches a validation loss of 0.0696. Real-time sign language translation receives an improvement through this proposed model which facilitates more accessible communication along with more efficient and interpretable processing for sign language communication.