Development of an Animation-Based Indian Sign Language e-Learning App for Persons With Hearing and Speech Impairment

Human-computer interaction (HCI) is changing the quality of our lives. In this paper, we present one such application of HCI that comes as a boon in the life of a hearing-impaired person. We propose to build an e-learning tool for the education of the hearing- and speech-impaired community by creating animated gestures in sign language, with particular emphasis on the Indian Sign Language (ISL). With the help of MATLAB App Designer, we developed a standalone MATLAB application for ISL fingerspelling (including digits, alphabets and combination of these two) and some ISL words of daily usage. This app can be used by the hearing-impaired community for learning ISL, even while seated at home, or for training signers and teachers at special schools. In our proposed method, the software first learns different sign gestures from real gesture videos. For this, hand regions in every frame are extracted using image processing algorithms and subsequently identifying key video frames that show significant change in hand shape and/or position during gesturing. Correlation coefficient and structural similarity are the two metrics used for the purpose. Next, several hand parameters such as the finger joint angles and orientation of the palm in these key frames are derived using Google MediaPipe holistic model. When using the app, for a query sign, the corresponding gesture is animated by synthesizing the key frames containing 3D hand shapes with desired palm orientation and finger joint angles using the stored hand parameters, followed by interpolating the in-between frames using image metamorphosis. For interpolation, we have used spline interpolation and intensity interpolation. Some sign gestures created in our experiments show that our proposed method generates smooth and natural-looking animated videos from a comparatively less amount of stored gesture information, thereby offering large savings in memory.