Generative semantic-enhanced vision-language pre-training for sign language translation

Vision-Language Pre-training (VLP) provides a promising way to acquire language-guided visual representations for Sign Language Translation (SLT). However, the unique temporal characteristics of sign language videos hinder the effective utilization of text supervisory signals. In this paper, we re-examine the impact of different text data on guiding visual feature extraction in VLP, demonstrating that text more temporally aligned with the videos yields more informative supervision. Based on this finding, we propose a Context-aware Dynamic Semantic Enhancement (CDSE) method to enhance text supervision. CDSE enables text data to simulate the temporal characteristics of the videos by integrating data-driven statistical properties with an adaptive expansion mechanism. Moreover, to reduce text annotation costs, we inject sign language priors into Large Language Models (LLMs) to generate pseudo-gloss sequences from spoken language sentences. We further design a multi-positive contrastive learning method to mitigate the adverse effects of individual low-quality pseudo-glosses. Extensive experiments on two widely used datasets, PHOENIX14T (8257 videos) and CSL-Daily (20,654 videos), demonstrate that the proposed method significantly outperforms the baseline. On the test sets of both datasets, our method improves BLEU-4 by 2.53 and 2.82 points, and ROUGE by 2.48 and 2.08 points, respectively.