Bridging Communication Barriers: Indian Sign Language to Multilingual Text and Audio Using Convolution Neural Network

Effective communication is essential in everyday life, particularly education, employment, and exchanging ideas and emotions. While speech is most people’s primary mode of communication, individuals with disabilities who cannot speak or hear often rely on sign language. This creates a barrier in communication. In this work, we propose a novel approach to convert Indian Sign Language (ISL) into multilingual text and audio, addressing a gap not covered by previous research. We curated a unique dataset of 20 English words, with five different demonstrations per word, resulting in 30,000 frames. Each video lasts 10 seconds and is recorded with either a white or black background at a resolution 1080p and 30 frames per second. The model leverages MobileNetV2, which has been customized to meet the specific requirements of sign language recognition. Our model achieves an impressive accuracy of 90%, enabling the conversion of ISL into multilingual text and audio, a significant advancement over previous work that typically focused on isolated language translation.