Speech-to-Sign Language Translation Framework for Telugu Language using Residual Bidirectional LSTM with Attention Network

The Speech-to-Sign Language Translation (S2SLT) system for the Telugu language is developed to improve communication between hearing individuals and the hearing-impaired community. Telugu, a widely spoken language in India, currently lacks adequate support in assistive technologies, especially for translating speech into Indian Sign Language (ISL). This work aims to build an intelligent system that captures spoken Telugu, processes it using Automatic Speech Recognition (ASR), and translates the recognized text into ISL using Natural Language Processing (NLP) techniques. Special attention is given to preserving the meaning and context during translation, as Telugu and ISL differ significantly in grammar and sentence structure. The paper proposes the design and implementation of a model that translates live voice or audio recordings of a native Indian regional language (Telugu) into Indian Sign Language (ISL) animations. In the initial phase, essential audio signals are collected from benchmark resources. From the collected audio, dual features such as spectral features and Mel-Frequency Cepstral Coefficients (MFCC) are extracted to represent the speech data effectively. These extracted features are passed into the speech-to-sign translation module. Here, a Residual Bidirectional Long Short-Term Memory with Attention Mechanism (Res-BiLSTM-AM) is employed to detect the corresponding sign text. The predicted sign text is then matched with a pre-defined Graphics Interchange Format (GIF) dataset containing ISL animation files. If a match is found, the corresponding GIF is generated as the final output. In cases where the predicted text does not match any entry in the GIF dataset, a Graphical User Interface (GUI) is used to display the ISL text output alphabetically, ensuring continued accessibility.