A Class‐Prior‐Adjusted Temporal Multi‐Expert Network for Word‐Level Lip Reading

ABSTRACT Lip reading aims to recognize speech from visual lip movements, requiring models to capture subtle variations in lip shape and complex temporal dependencies. Compared with sentence‐level lip reading, word‐level lip reading offers limited linguistic and semantic context and therefore depends more heavily on fine‐grained, short‐term motion cues. Many existing methods use a shared visual backbone followed by a single temporal modelling branch and a unified classifier. This single‐path decision architecture may limit their flexibility in modelling heterogeneous word‐level temporal dynamics. To address this limitation, we propose a class‐prior‐adjusted temporal multi‐expert network (CP‐TME) for word‐level lip reading. CP‐TME employs a two‐stage training strategy that separates shared visual representation learning from multi‐expert temporal decision modelling. In the first stage, we adopt PaCo‐based contrastive pre‐training, which combines learnable parametric class representatives with a dynamic labelled feature queue, to learn discriminative shared visual representations. In the second stage, the pre‐trained VideoCNN backbone is frozen, while multiple experts with independent temporal branches and classification heads are trained using class‐prior logit adjustments derived from empirical class frequencies. Negative and positive prior exponents impose opposing class‐wise logit shifts, whereas the zero exponent preserves the unadjusted logits. These three conditions produce different class‐wise probability distributions and branch‐specific optimization gradients, driving the experts toward distinct temporal decision functions. CP‐TME achieves recognition accuracies of 60.37% on LRW‐1000 and 91.81% on LRW, demonstrating its effectiveness for word‐level lip reading.