Visual Speech Recognition with Surrounding and Emotional Information

Visual Speech Recognition has made significant progress, with the current mainstream approach focusing on extracting lip features and using deep learning to enable the model to understand a speaker’s content from video alone. However, one might question whether visual language recognition is synonymous with lipreading. Whether we can extract additional information beyond lip movements to improve model performance. In this study, we developed the Lip-Face-Surrounding model to comprehensively extract information from video. Our findings reveal that the information beyond the lips— such as eye corner movements, jaw movements, nostril movements, throat movements, and shoulder movements—are captured by the model and serve as discriminative features for visual speech recognition. This information is also applicable to handcrafted datasets. Additionally, indirect information such as the speaker’s body language and interactions with the surrounding also impact model performance. This information provides extra context that enhances performance, while other times, it introduces noise that affects model convergence. Our model achieved promising results on the CN-CELEB and GRID datasets, with a 5% absolute performance improvement over the lip-only approach.