A Graph-enhanced Multimodal Transformer Framework for Continuous Sign Language Recognition and Gloss-assisted Translation
Continuous sign language translation (CSLT) aims to convert continuous signing into spoken-language text, but fixed multimodal fusion may fail to adapt to changing visual cues.We propose a graph-enhanced multimodal Transformer using gloss-assisted training with gloss-free inference.The framework combines RGB appearance, skeletal pose, and facial landmarks through adaptive cross-modal weighting and a temporal saliency gate.A CTCbased recognition objective and an autoregressive translation objective are jointly optimized during training, while gloss prediction is not required during inference.On PHOENIX-2014T, the model achieved 17.80 ± 0.20% WER, 25.40 ± 0.12 BLEU-4, and 34.80 ± 0.15 ROUGE-L.On CSL-Daily, it improved BLEU-4 by 0.70 over the strongest baseline.The framework achieved 22 FPS inference throughput under the reported configuration, while ablation and robustness experiments indicated contributions from multimodal fusion and resilience to degraded inputs.
Authors
- Asha K H
- Chaithra A S
Institutions
- Don Bosco Institute of Technology (IN)
- Visvesvaraya Technological University (IN)
Publication Details
- Journal
- International journal of intelligent engineering and systems
- Published
- 2026-11-17
- DOI
- https://doi.org/10.22266/ijies2026.1231.24
- Primary Topic
- Hand Gesture Recognition Systems
- Type
- article
- Field-Weighted Citation Impact
- 0.00