Deep Spatio-Temporal Networks for Visual Speech Recognition: A Hybrid CTC-Cross Entropy Approach

Visual Speech Recognition (VSR) is a challenging task that involves interpreting spoken language by using visual cues from lip movements. This paper presents an end-to-end deep learning framework for lipreading, leveraging a combination of 3D Convolutional Neural Networks (CNNs) and Bidirectional Long Short-Term Memory (BiLSTM) networks. The model processes grayscale video frames, extracts spatiotemporal features using 3D CNN layers, and models temporal dependencies using BiLSTM layers. A hybrid loss function combining Connectionist Temporal Classification (CTC) and Categorical Cross-Entropy (CE) was used to improve the robustness of phoneme sequence alignment.