A Real Time Deep Learning Model for Sign Language Interpretation

This work presents a system based on a Convolutional Neural Network (CNN) that can transcribe American Sign Language (ASL) gestures to text. The significance of ASL for the Deaf and Mute (D&M) community cannot be understated but there are many barriers to usage and adoption of ASL. Through the use of computer vision and artificial intelligence, the present work extends the usability of ASL for users who happen to be deaf or hard of hearing by offering real-time translations. Using input from a usual camera, the system preconditions the gesture image by employing a Gaussian blur to minimize noise, cropping it, and converting it to grayscale. After being trained on a customized dataset that included 26 distinct static hand gestures that represented the alphabets in ASL, the CNN performs exceptionally well, achieving 99.57% test accuracy, 99.59% precision, 99.57% recall and 99.56% F1 score. To increase usability, a text-to-speech module is added for auditory feedback. Overall, the proposed work poses as a step toward a scalable, low-cost, and fit-to-average accuracy gesture recognition system on standard consumer devices such as a laptop and even a smartphone. It shows the powerful potential of machine learning and computer vision to make the world more inclusive by bridging the communication gap between hearing and non-hearing people in the D&M community.