Interpretation of Indian Sign Language to Text and Speech to Communicate with Speech and Hearing Impaired Community

One of the necessities of a human being is to express themselves. To express one’s thoughts, people typically use speech. However, individuals who are deaf and with speech-impairment convey their feelings by practicing sign language. With this communication medium they perform gestures and interpreting these gestures means knowing sign language(SL). Everyone is not acquainted with SL which is the main obstacle in this communication process. To address this key issue, we recommend the deep learning-based approach which assists in interpreting Indian Sign Language(ISL) to text and speech. Communication between a hearing-impaired individual and an illiterate/blind person is made possible through speech conversion. We created a sample model which interprets numbers used in ISL. Numbers are helpful at many places; financial institutions like banks, cash counters, to mention time/distance. Existing sign language recognition systems do not adequately address the challenges posed by i) left, right and ambidextrous signer ii) signers facing mobile camera in selfie-mode iii) signers wearing gloves on their hands to facilitate communication in cold climates or at hill stations iv) If more than one gesture is present for particular sign then those should be considered. To address the mentioned problems, we have created a diverse dataset named ISL_Num14_AS3N24. We have proposed a system wherein we made use of deep learning technique along with our suggested methodology which utilises the hand keypoints in the pre-processing step. Also, for quicker training, we employ modified two pretrained models Vgg16 and MobileNetV2. Text-to-speech conversion module utilizes gtts library. The system exhibits excellence performance with validation accuracy for Vgg16 as 99.76 and for MobileNetv2 as 99.44. Furthermore, the validation loss of VGG16 and MobileNetv2 is 0.9471 and 1.9921 respectively.