Real-Time Sign Language Interpretation and Translation to Speech Using Cuda and Machine Learning

This paper introduces a novel deep learning-based system for real-time American Sign Language (ASL) interpretation and translation into speech, aimed at improving communication for individuals with hearing and speech impairments. The system integrates MediaPipe, a state-of-the-art framework for real-time hand and body pose tracking, with a custom deep learning model designed to recognize and classify ASL signs. Video frames are processed to extract precise hand landmarks, which are then interpreted by the model for context-based sign classification. CUDA parallelization is utilized for efficient, high-performance processing, and the system is optimized with TensorRT and cuDNN for real-time GPU acceleration. The recognized ASL signs are then converted into naturalsounding speech through a text-to-speech (TTS) module. To ensure grammatical accuracy and contextual integrity, the system employs Generative AI (Google Gemini Pro open-source model) to refine the translated text before speech synthesis, enhancing the naturalness of the output. The proposed system offers scalability, robustness, and accuracy, making it suitable for applications in accessibility, education, and public services, thereby facilitating seamless communication between ASL users and non-users.