AI-Driven Gesture Recognition and Multilingual Translation

Abstract – This paper presents a real-time system designed to improve communication effectively for everyone with speech and hearing impairments through gesture-based language translation. This approach uses machine learning algorithms to interpret American Sign Language (ASL) hand gestures which is a universal sign language and convert them into both text and speech outputs. By integrating Mediapipe for landmark detection with a Convolutional Neural Network (CNN) for gesture classification, the system effectively identifies static hand signs and ensures robustness in diverse surrounding environment with proper lighting conditions. The recognized gestures are further translated into various languages using natural language processing techniques, followed by speech synthesis through text-to-speech tools. The implementation includes a user-friendly graphical interface, enabling live gesture capture and multilingual feedback, making the system both accessible and adaptable for real-world scenarios.