Deafy: An AI-Powered Bidirectional Communication System for Real-Time Sign-to-Audio and Audio-to-Sign Translation

The increasing advancements in Artificial Intelligence, browser-based inference, and computer vision have enabled new possibilities for inclusive and accessible communication technologies. Existing sign-language interpretation systems are typically limited by static datasets, lack of real-time performance, complex setup requirements, or dependency on specialized hardware. This project introduces Deafy, an advanced, unified, and browser-based communication platform that enables seamless bidirectional interaction between hearing and hearing-impaired individuals. Deafy integrates two core modules — Sign-to-Speech and Speech-to-Sign — allowing real-time gesture recognition and speech-based translation directly within the browser. The system leverages MediaPipe Hands for high-precision hand-landmark extraction, TensorFlow.js for on-device gesture classification, and the Web Speech API for speech recognition and synthesized audio output. The frontend is built using React + Vite, ensuring a responsive, installation-free interface accessible to any user with a modern browser. The architecture is structured into three layers: (1) a user interaction layer built with modular React components for accessibility and instant navigation, (2) an application logic layer performing real-time hand tracking, gesture classification, speech-to-text, and text-to-speech operations, and (3) an output layer delivering translated results through voice, captions, and animated sign videos. By eliminating installation barriers, processing all AI models in-browser, and offering intuitive interaction for both communication directions, Deafy demonstrates a scalable, efficient, and inclusive model for bridging accessibility gaps using AI-driven interaction.