Sign Language Translation using Hybrid Temporal Convolutional Network with TTS Fusion

Sign language recognition and translation into spoken language remains a complex and evolving challenge due to the limited availability of large-scale datasets and the intricate temporal patterns of sign gestures. This work presents a novel approach that integrates Temporal Convolutional Networks (TCN) with Text-to-Speech (TTS) fusion to enable real-time translation of Indian Sign Language (ISL) into natural-sounding speech. The primary objective is to develop an efficient and accessible system that bridges the communication gap between hearing-impaired individuals and the broader community. The proposed model is trained using the ISLRTC dataset, which contains a diverse collection of ISL videos. The system comprises three core components: sign language recognition, text generation, and TTS fusion. Initially, relevant features are extracted from input sign language videos using computer vision techniques and passed through a Convolutional Neural Network (CNN). These features are then processed by the TCN model, which is well-suited for sequential data and effectively captures the temporal dependencies of sign gestures. Once the gestures are recognized, the corresponding text is generated and subsequently converted into speech using a TTS system. To enhance the auditory experience, the speech output is fused with the original video by synchronizing audio with visual frames and adjusting pitch and volume based on visual cues. This fusion not only reinforces the communication intent but also delivers a more natural and engaging user experience. The approach offers advantages such as improved sequential modeling through TCN, enhanced speech output through TTS fusion, and efficient performance even with smaller datasets, making it suitable for real-world deployment in assistive technologies.