Consistency-Driven Cross-Modality Transferring for Continuous Sign Language Recognition

Sign language consists of a unique grammar and expression system. While Continuous Sign Language Recognition (CSLR) approaches share similarities with conventional NLP approaches in which language understanding is involved, the current works on CSLR usually focus on feature extraction and fusion, neglecting the language semantics, causing false positives and overfitting in gloss detection. In this paper, we propose a novel Consistency-Driven Cross-Modality Transferring (CDCM) mechanism to transfer the language modality to visual modality under a consistency-driven optimization. By progressively reducing the gap between text and visual modality, we are able to stably train CSLR networks. The experiments show the efficacy of our approach, with a notable relative reduction in Word Error Rate of 6.91% on average across multiple datasets. We also demonstrate that our approach contributes corrections to suppress the false peaks on highly related and visually similar glosses while training, making glosses in semantic space distinct, thereby achieving improved overall performance.