Efficient Mouth Alignment for Visual Speech Recognition

Visual Speech Recognition (VSR), also known as lip-reading, requires accurate mouth tracking and alignment to achieve high performance. Conventional approaches rely on complex pipelines involving facial landmark detection, facial landmark smoothing, affine transformation estimation and warping. In this paper, we propose a streamlined solution: a fast Mouth Alignment Network (MAN) that directly estimates affine transformation parameters from a video of a talking face, significantly simplifying the process. Our approach also incorporates a complementary Mouth Scoring Network (MSN) that injects knowledge about adversarial perturbations into the training process, further improving alignment quality. Experiments on standard lip-reading benchmarks (LRS2 and LRS3) demonstrate that our method maintains the performance when integrated with state-of-the-art VSR systems, including Auto-AVSR and USR, while substantially improving the alignment quality and computational efficiency.