TriFuseNet: Open-Vocabulary Sign Language Translation via Hierarchical Image-Text-Keypoint Fusion

In the field of bridging communication gaps between deaf and hearing communities, sign language recognition (SLR) and sign language translation (SLT) systems face persistent challenges in cross-lingual contexts as the understanding ability of unlabeled data is limited, the fusion depth of traditional fusion modes is insufficient, and inadequate temporal modeling, which hinder practical performance. To address these limitations, this study proposes TriFuseNet, an innovative open-vocabulary tri-modal fusion framework grounded in intersemiotic translation theory, dynamic equivalence theory, and multimodal discourse analysis, which integrates RGB images, textual semantics and skeletal keypoints through a hierarchical interaction mechanism to achieve fine-grained cross-modal alignment. The framework introduces a vision-language interaction module to bridge visual features with linguistic structures, while a bilinear state-space model (BSSM) captures long-range temporal dynamics for gesture-linguistic coherence. Additionally, region-specific attention mechanisms prioritize critical body regions to model culturally specific gestures with nuanced spatial-temporal details. Experimental evaluations on the CSL-Daily and PHOENIX-14T datasets demonstrate TriFuseNet’s superiority over state-of-the-art methods, achieving 2.3% and 1.6% reductions in word error rate (WER), respectively.