EchoHands: A Deep Learning-Based Multimodal Gesture Recognition and Real-Time Speech Synthesis System for Deaf-Mute Communication

Abstract - Although few people know sign language, there is still a limited communication between the deaf-mute community and the hearing population. In this paper we propose EchoHands, a deep learning based multimodal system that bridges this gap with two complementary pipelines: a sign-to-text-to-speech pipeline that records hand gestures using a webcam, extracts landmark features with MediaPipe, and classifies them using a Convolutional Neural Network, then transforms the recognized text to audible speech; and a text and speech to sign pipeline which maps a typed or spoken input to a sequence of sign gestures via an avatar-based animation module. The system is implemented as a full-stack web application with Flask backend and React.js frontend that enables real-time interaction via browser without the need for hardware like gloves or depth sensors. The recognition model obtained an overall accuracy of 97.31 percent on a held-out validation set of 260 images belonging to 26 gesture classes. The macro-averaged precision, recall and F1-score are 97.82 percent, 97.35 percent and 97.48 percent respectively, and average inference latency of 110 to 135 ms per frame on standard consumer CPU hardware. The text-to-sign pipeline produces recognizable animated sequences for short phrases.