A Novel Sentence‐Level Visual Speech Recognition System for Vietnamese Language Using ResNet3D and Zipformer

This paper presents the first sentence‐level visual speech recognition (VSR) system specifically designed for the Vietnamese language. We have developed a unique dataset comprising 115 h of video recordings from over 100 speakers, focusing on single‐speaker scenarios. The proposed VSR system utilizes a ResNet3D architecture as the visual frontend, paired with a neural transducer framework featuring a Zipformer speech encoder. It incorporates a stateless decoder that considers two preceding tokens and is optimized with a pruned‐RNNT loss function. Experimental results show that our system achieves a word error rate (WER) of 27.14% and a character error rate (CER) of 20.45% on single‐speaker tasks, demonstrating significant progress in VSR for Vietnamese.