Cross Attentive Multi-Cue Fusion for Skeleton-Based Sign Language Recognition

Sign language, primary communication medium of the Deaf, uses visual cues from upper body, hands and face. Sign Language Recognition (SLR) aims to learn salient representations from these cues to bridge the communication gap between Deaf and hearing communities. The state-of-the-art Graph Neural Network-based SLR frameworks often transform sign videos into a sequence of graphs formed by hand and body joints. We show that learning sign representations using only body topology results in suboptimal solutions. We also show that training the graph-based SLR frameworks with a domain-specific hand topology can single-handedly help reach state-of-the-art recognition performance. Therefore, fusing the representations learned from the topologies of multiple visual cues is essential to a generalizable and robust SLR framework. However, integrating multiple cues in an SLR framework becomes challenging due to the difference in the spatial and temporal dynamics of hand and upper body movements. This study exposes the differences in the temporal patterns of hand and body representations extracted from the Spatio- Temporal Graph Convolutional Network (ST-GCN) using self-attention. Considering these distinctions, we propose a multi-cue cross-attention framework to enable hand and body cues to attend to each other differently while fusing their representations for SLR. Thus, the proposed attention-based fusion framework allows us to explain how individual articulators affect the model’s predictions for correctly classified and misclassified signs. Our experiments on two benchmark isolated sign language datasets, BosphorusSign22k and AUTSL, show that the proposed framework’s performance of isolated SLR is on par with state-of-the-art while highlighting the benefit of choosing domain-specific hand graph topologies and fusing multiple cues for SLR. Furthermore, our cross-attentive approach for fusion of upper-body and hand cues improves recognition accuracy by around 1% and 3% on the respective datasets over hand-only models, while making the SL recognition interpretable, and demonstrates the complementary interactions between visual cues.