Khasi Sign Language Recognition using Google’s Mediapipe and Deep Learning Feedforward Neural Network Approach

Sign language is utilized by individuals who have difficulty hearing or speaking to communicate. Developing technology-assisted tools for interpreting sign language is critical, especially for regional languages such as Khasi Sign Language in India. This would significantly improve communication in indigenous languages. While substantial studies have been conducted on numerous world languages, there is still an opportunity to develop tools and strategies to support local Indian sign languages. This work attempts to utilize the advancement of deep learning to create a system capable of identifying gestures made by the hands in the low-resource Khasi Language. A 14,49,252 data point set was created to represent 18 static two-hand gestures related to the Khasi alphabet. These 18 static signs were derived from the alphabets of the Khasi language, including ‘A’, ‘B’, ‘D’, ‘E’, ‘G’, ‘I’, ‘Ï’, ‘K’, ‘M’, ‘N’, ‘Ñ’, ‘NG’, ‘P’, ‘R’, ‘S’, ‘T’, ‘U’, and ‘W’. Prior to extracting the hand landmarks using Mediapipe, we enhanced the raw image using the Contrast Limited Adaptive Histogram Equalization (CLAHE) method. This approach improves the visibility of images that are underexposed or lack contrast. A feedforward neural network model was developed for 18 signs of the Khasi alphabet. The proposed model attained an overall accuracy of 99% on the 3451 test images. This innovative approach can potentially revolutionize the communication barrier for individuals with speech and hearing difficulties in indigenous Indian languages by addressing a crucial need in the community.