Multi-scale Variable Window Attention Network for Continuous Sign Language Recognition

Sign language is primarily conveyed through hands and face movements. Therefore, a pivotal challenge in continuous sign language recognition (CSLR) is the meticulous modeling and interpreting of visual cues emanating from hands and facial movements. However, contemporary CSLR methods have exhibited limitations in adequately capturing information from both the hands and facial regions, often overlooking the continuity, and correlations across multiple consecutive frames. To mitigate these shortcomings, we propose the Multi-scale Variable Window Attention Network (MVAN). This framework fuses the strengths of an improved Efficient Multi-scale Attention (iEMA) with Variable Window Adaptation (VWA) attention. Specifically, iEMA first scans information spatially at multiple scales to simultaneously capture information from areas such as hands and faces, and models spatial continuity interactions between them, enhancing spatial features. Secondly, VWA dynamically changes its window size at the temporal level to adaptively capture complex temporal changes in hands and face movements in successive frames. Thus, VWA can effectively model the continuity and correlation between hands and face movements to have the ability to capture sign language movements of different lengths. Comprehensive experimental results demonstrate that our proposed MVAN strengthens the visual feature extraction of hands and face, improving the recognition performance.