SignSpeak – Sign language translation system for hearing impaired

The deaf and hard-of-hearing community face significant challenges navigating through daily-life. Difficulties such as hearing impairments or speech disabilities often limit communication for the community which is a crucial aspect of human life. To bridge this gap, Sign Language, is a methodology that involves hand movements with face interaction that acts a medium to converse is utilized to convey the ideas that a hard-of-hearing wishes to share. However, Sign Language exists in different accents and does not follow a universal common language and is quite uncommon for an average group of people to have knowledge base regarding it. There are translators that have Sign Language understanding that interpret information to the hearing-impaired via hand signs. But they exist few in numbers along with their availability varying at times. In such situations, hearing-impaired community cannot actively participate in interactions. SignSpeak, a real-time translation system is developed that records audio input from the user and provides the transcribed sign gloss clips that interprets the user's vocabulary in sign language. Bidirectional Encoder Representation from Transformers (BERT) - a deep learning algorithm combines PyAudio to obtain audio input with OpenAI's Whisper model to generate text transcription and reorder them. It generates tokens which are then read by the FFmpeg module for sign gloss video retrieval and stitching and present a complete video output.