Advancing Chinese Lip Reading through Contextual Enhancement

Lip reading, also known as visual speech recognition, aims to extract spoken words from silent videos by analyzing lip movements. Most existing methods have been developed primarily for English-language videos and are not optimized for Chinese lip reading. Given that Chinese is rich in homophones, which demand nuanced contextual cues for accurate disambiguation, conventional lip reading techniques tend to experience a significant drop in performance when applied to Chinese. To address these challenges, we propose the Context-Enhanced Visual Speech Recognition Network (ConVSR), specifically designed to improve Chinese lip reading. ConVSR incorporates intermediate connectionist temporal classification (inter CTC) residual modules to provide enhanced CTC supervision, improving the model’s contextual comprehension. Additionally, a Bi-Transformer Decoder is employed to simultaneously capture both past and future contextual information, thereby strengthening the model’s semantic understanding. Our approach achieves a $74.40 \%$ character error rate (CER) on the CAS-VSR-MOV20 dataset in track 1 of the Mandarin Audio-Visual Speech Recognition Challenge (MAVSR) 2025, securing first place and underscoring the effectiveness of ConVSR in advancing Chinese lip reading accuracy.