Phoneme-Level Visual Speech Recognition via Point-Visual Fusion and Language Model Reconstruction

Visual Automatic Speech Recognition (V-ASR) is a challenging task that involves interpreting spoken language solely from visual information, such as lip movements and speaker-dependent variance. A major difficulty arises from viseme ambiguity, where multiple phonemes share similar visual appearances, making word- or character-level prediction errorprone. To address this, our work focuses on phoneme-level modeling, which provides a finer linguistic granularity than visemes while reducing the reliance on large-scale pretraining. We propose a novel phoneme-based two-stage framework that fuses visual and landmark motion features, followed by a Large Language Model (LLM) for sentence reconstruction to address these challenges. Stage 1 consists of Point Visual Automatic Speech Recognition (PV-ASR), which outputs the predicted phonemes addressing speaker-specific facial characteristics. Stage 2 comprises No Language Left Behind (NLLB) LLM, which reconstructs the output phonemes back to words. Our method demonstrates promising performance in terms of Word Error Rates (WER), especially on LRS2 (16.0% WER) while maintaining competitive results on LRS3 (20.3% WER).