Transformer-based motion-visual integrated fusion for isolated sign language recognition

Sign Language Recognition (SLR) is a critical technology that facilitates seamless interaction between hearing-impaired individuals and the broader community. Among the various tasks in SLR, isolated sign word recognition remains particularly challenging due to high visual similarity between signs and variability in hand motion dynamics. To address these challenges, we propose Transformer-based Motion and Visual Integrated Fusion for Isolated Sign Language Recognition (TransMoVIF), a novel dual-stream transformer-based framework that effectively combines skeletal motion trajectories and visual appearance cues for enhanced recognition. The proposed model extracts fine-grained motion signatures by encoding trajectory-level features—such as velocity, displacement, and curvature from hand keypoints. Simultaneously, a visual stream processes RGB keyframes identified using a crucial frame selection algorithm based on pose variations and structural similarity. These two complementary modalities are fused using a cross-attention mechanism, allowing the model to learn intricate relationships between motion dynamics and visual features. Unlike conventional late-fusion methods, TransMoVIF enables deep integration of semantic and temporal patterns across streams, enhancing its ability to differentiate between visually similar signs with distinct motions. Extensive experiments on two publicly available isolated SLR datasets demonstrate that TransMoVIF outperforms state-of-the-art unimodal and fusion-based models in terms of both accuracy and robustness, establishing a new benchmark for isolated sign language recognition.