Deep Learning and Computer Vision based Smart Intercom Device for the Deaf and Mute People

Effective communication remains a significant challenge for individuals who are deaf or hard of hearing, particularly when relying on traditional verbal methods. To address this, we propose a multimodal intercom system that integrates voice detection, gesture recognition, lip reading, audio-to-text conversion, and real-time feedback to enhance communication between deaf and hearing individuals. The system utilizes a microphone to detect specific voice commands and activate a vibration motor to alert users. A camera, combined with computer vision and deep learning techniques, facilitates gesture recognition and lip reading, transcribing movements into text for seamless interaction. Additionally, the system processes audio from video calls to provide real-time speech-to-text transcription, enabling deaf users to follow conversations effectively. Gesture-based communication is supported through hand gesture recognition, translating gestures into text or voice commands displayed via text-to-speech functionality. By combining these features, the proposed system improves accessibility and fosters independence, providing an inclusive and adaptive solution to bridge communication gaps between the deaf and hearing communities.