Enhancing Lip Reading for the Hearing Impaired using CNN AND LSTM

Speech recognition through lip reading can help people with hearing loss, especially in noisy environments. Traditional communication tools are not designed to handle the needs of people who are deaf or hard to hear. This study developed a method that combines short term neural networks (LSTM) with convolutional neural networks (CNN) to record spontaneous words based on the appearance of lips. It involves preprocessing video frames, including resizing and filling with black frames when necessary, and using transformations to increase the predicting ability of the model. This model uses Time Distributed CNN layers and bidirectional LSTM layers to identify the time dependent physical properties of the image. This integration allows the system to correctly interpret and convert movement of lip into text. The model is trained using a custom data generator application that generates augmented data. The results of this study show significant improvements in lip reading accuracy upto 95%. People with hearing loss can independently understand more of the spoken content without the constant assistance of an interpreter or translator.