CNN LIPNET : Automated Lip Reading Using Deep Convolutional Neural Networks

Lipreading, or the process of interpreting the speech by analyzing lip movements, is a critical solution when auditory cues are limited or even unavailable. This project introduces the LipNet, a vision-based neural network that uses CNNs and LSTM networks to decode lip movements into text. The model is trained on the Grid Corpus dataset, with data split into 80% for training and 20% for testing. Keyframes extracted from video sequences are processed to capture both spatial features using CNNs and temporal patterns with LSTMs. A SoftMax layer outputs accurate textual representations of speech. Achieving a notable 97.44% average accuracy, LipNet demon- strates robust performance across different speakers, accents, and speaking styles. It has broader implications on human- computer interaction, especially in assistive technologies for the hearing impaired, environments with high noise interference, and silent communication, among others. The model improves focus on crucial visual cues through the integration of attention mechanisms.