Indian Sign Language Recognition System Using Deep Learning

Communication is an essential part of human interaction. At the same time, people with hearing and speech impairments encounter many obstacles in expressing their thoughts and receiving information from others. Indian Sign Language (ISL) is the main tool for communication of people who are deaf and hard of hearing in India. Nevertheless, the vast majority of the common population does not know how to communicate through ISL. It creates a huge communication gap for the deaf and hard of hearing community that affects their education, job search, access to healthcare services, etc. This paper introduces the Indian Sign Language Recognition System based on the use of deep learning and computer vision methods to automatically recognize ISL gestures in real-time. The system is based on the landmark-based approach. Google MediaPipe's Hand Landmarker model is used to get 21 three-dimensional hand keypoints (63 features) per each frame and classify them using Keras Multi-Layer Perceptron (MLP). This two-steps method is translation-invariant, scale-invariant, and computationally inexpensive and allows for real-time inference on CPU-only hardware without the need for GPU acceleration. Any number of ISL gesture classes can be defined by the user. Experiments show that the landmark-based MLP model has high validation accuracy (usually around 93-97% for 5-10 ISL classes) and less than 10 ms latency per frame. This result is achieved much faster than CNN-based pixel classifiers. The system works on a MacBook or laptop computer with webcam.