Parameter-Efficient Personalization of Foundation Speech Models: LoRA Adapters for Visual Speech Recognition

Visual speech recognition (VSR) models trained on thousands of hours of public video still make many errors on any particular person. We ask whether a small, person-specific adapter, trained on a few minutes of that person's ordinary webcam-style speech and labelled automatically by an audio speech recognizer, can reliably reduce those errors. Using a public Auto-AVSR visual-only model (251.6M parameters) and 12 English speakers from public talking-head videos, we train low-rank adapters (LoRA; 1.18M parameters, 4.7 MB) on Whisper-generated transcripts and evaluate each speaker on an entire held-out recording session. With 15 minutes of own-speaker video, LoRA reduced word error rate (WER) for all 12 speakers, by 2.91 points or 11.4% relative on average (one-sided Wilcoxon signed-rank p = 2.4×10⁻⁴; speaker-stratified permutation p < 10⁻⁴). The effect was hypothesized from a four-speaker pilot and then replicated, with hyperparameters unchanged, on eight new speakers (8/8 improved, 12.5% relative, p = 0.004). Gains grew with more own data rather than overfitting: training on all available own-speaker video (17–75 minutes) improved further for 9 of 12 speakers and reached 15.8% mean relative reduction (up to 25%), exceeding full fine-tuning on 15 minutes (14.4%) while storing over 200× fewer parameters. Adapters were person-specific: in 50 sampled owner/non-owner comparisons, the owner's adapter was better in 48, with no significant reversals. A pilot ablation places the gain mainly in the visual encoder rather than the text decoder, and shows that plain full fine-tuning degrades accuracy at 2–5 minutes of data while LoRA does not. Timing in a virtualized Apple M4 Pro environment suggests an adapter can be trained on a laptop-class machine in well under an hour. All labels are automatic and all speech is voiced; we discuss implications and limits for silent-speech interfaces.