Context-Aware Lip Reading and Speech Generation Using Deep Learning

Visual speech recognition is challenging under limited supervision and CPU-only deployment constraints. Thispaper evaluates an isolated-word recognition pipeline that combines landmark-based mouth registration, a frozen AV-HuBERTBase encoder, temporal statistical pooling, and a regularizedmultinomial classifier. The experiment defines an eight-wordsubset of MIRACL-VC1 with speaker-separated developmentpartitions and a quarantined pilot subset. Of 720 designatedtraining utterances, 704 pass tracking checks; 240 validationand 192 test utterances are retained. Validation selection over27 layer–pooling–regularization configurations identifies layer-9mean–standard-deviation features. On the specified test utterances, the selected pipeline correctly recognizes 182/192 words(94.79% top-1 accuracy; macro F1 0.9467), compared with 24/192for a historical 3D-CNN–BiGRU system. This 82.29-percentagepoint difference is a whole-system comparison, not a controlledarchitectural ablation: preprocessing and supervised training-setsizes differ. Results establish limited-vocabulary recognition onthe audited protocol, not continuous transcription, contextualcorrection, or synthesized speech.