A Robust Multimodal ISL Translation Framework

Sign language is vital for bridging the communication gap between the deaf community and the hearing population. However, the widespread lack of familiarity with Indian Sign Language (ISL) among most people leads to a significant disconnect. To address this, a robust ISL recognition system using computer vision technology is being developed to interpret ISL gestures, encompassing both static signs (alphabets and numerals) and dynamic signs (words and sentences). Effective communication between ISL users and the broader community is hindered by the absence of reliable ISL translation systems, mainly due to the lack of appropriate datasets. Recognizing ISL is challenging due to the variability in signs, the use of both hands, and the fluidity of hand movements. This work aims to create an effective Augmentative and Alternative Communication (AAC) system capable of understanding continuous Indian sign language. By utilizing hand, face, and body movements as multimodal features, the system strives to achieve comprehensive ISL translation. A custom dataset has been generated by stitching videos of dynamic ISL words. The system uses a Bi-LSTM with Connectionist Temporal Classification, which is effective in ISL translation. With a word error rate of 0.12 and a sentence-level accuracy of 54.37 %, this research highlights the transformative potential of technology in promoting inclusivity and enhancing communication for the hearing-impaired community.