Enhanced Realtime Sign Language Detection and Multilingual Translation System

Sign language is the communication medium that is utilized by the hearing-impaired community. The medium is visual and it uses hand signs and symbols for communication. Over 300 different types of sign languages present in the world today. Sign language uses gestures and hand movements for the majority of the words and finger spellings for the rest. Nonverbal speakers face a lot of challenges while interacting with verbal speakers. Even non-verbal speakers who speak different sign languages face difficulties communicating with each other. This project proposes an enhanced real-time system which detects gestures and finger spellings of multiple sign languages, and translates them into multiple spoken languages. The proposed system aims to utilize CNN network for finger spelling recognition and Long Short-Term Memory (LSTM) network for gesture interpretation. The system will use computer vision for accurate gesture detection and Google Translator API for recognizing signs which will be translated into multiple spoken languages. Initial tests with isolated datasets indicate promising accuracy for both finger spelling and gesture recognition. The goal of the completed system is to help aid communication between verbal and non-verbal speakers and to provide an efficient mode of communication between the two communities.