Multilevel Conversion of Indian Sign Language from Gesture to Speech

Abstract: Indian Sign Language (ISL) serves as a primary mode of communication for Deaf and hard-of-hearing communities in India. However, despite its societal importance, ISL remains largely unsupported by mainstream technological platforms, limiting inclusive communication. This research introduces a real-time ISL recognition andtranslationsystem thatconvert shandgesturesintocorresponding text and speech outputs, enabling phrase-level communication rather thanisolated characterinterpretation. The architecture uses a modular pipeline approach, with a Convolutional Neural Network (CNN) for accurate gesture classification, a phrase-mapping module to translate gestures into meaningful expressions, a MediaPipe for accurate hand landmark detection, and a text-to-speech (TTS) system to turn the generated text into audible speech output. Unlike previous systems restricted to static signs, our approach supports semantically rich, multi-word phrases, enhancingnatural communicationflow. Aspecially constructed dataset of ten frequently used Indian Sign Language (ISL) phrases was used to train the model. To improve generalization,150samplesfromeachclassweretakeninvarious lighting and background conditions. The final system achieved 95% classification accuracy, operated at 60 frames per second, and maintainedlatency below100milliseconds. Usabilitytestingwith multiple users confirmed the system's robustness, responsiveness, and accessibility. The findings demonstrate the viability of deploying deep learning-based ISL recognition systems in authentic environments, including public areas, healthcare facilities, and educational institutions.