Robust Lipreading Through a Dual-Stream Adaptive Framework with Spatiotemporal Landmark

Lipreading, as a critical tool for interpreting human intentions in noisy or silent environments, faces performance bottlenecks due to speaker variability. To mitigate this issue, we introduce a dual-stream adaptive architecture that integrates spatiotemporal lip landmark features and task-specific adaptation techniques to enhance the model's adaptive capabilities. Specifically, we design a dual-stream framework that fuses lip landmark and visual features, incorporating dual-stream adaptive strategy in both the landmark and the visual streams to model speaker-specific dynamics while preserving fundamental generalizable representations. Adapter modules are also employed to smooth the dual-stream representations and optimize the fusion quality. Experimental results on the word-level public dataset LRW-ID, demonstrate that our approach significantly enhances the model's speaker adaptation performance with stable parameter efficiency. Notably, with only 5 minutes of adaptation data, the recognition accuracy on LRW-ID reaches 91.25%. The proposed approach effectively enhances the adaptive performance of lipreading, significantly expanding the practical applicability.