Chinese word level lip recognition based on dual branch collaborative attention network

Lip language recognition, as a key language perception task, has shown great potential for applications in complex acoustic environments and public safety. However, existing Chinese word level lip recognition methods still face significant challenges in capturing subtle and rapid dynamic changes in the lips and distinguishing visually similar vocabulary. To address these challenges, this article proposes a novel approach called Lipreading with Dual Branch Collaborative Attention Network, LipDBCA. The core innovation of this network lies in its dual branch feature extraction front-end, which processes spatial texture details and spatiotemporal dynamic features in parallel. Specifically, one branch focuses on high-resolution spatial features, while the other branch is dedicated to capturing motion patterns across frames. In order to achieve efficient integration of information from two branches, this paper proposes a cascaded gating fusion mechanism that can purify the features of each branch and dynamically perform weighted fusion based on input. In addition, the model integrates multi-dimensional attention mechanisms of channels, space, and time to enhance the expression of key features and suppress noise. The backend of the model uses bidirectional gated recurrent units to model long-range temporal dependencies. The experimental results on the LRW-1000-Subset dataset containing 100 types of Chinese vocabulary showed that the proposed LipDBCA model achieved a macro average accuracy of 71.13%. These results strongly validate the effectiveness of the cross branch feature collaboration strategy and multi-dimensional attention mechanism proposed in this paper in handling complex lip language recognition tasks, proving that decoupling and intelligently integrating spatial and dynamic features is a superior recognition paradigm.