Zero-Shot Multi-View Australian Sign Language Recognition

Sign language is a common method of communication used by deaf individuals or those with speech impairments, possessing a complexity comparable to spoken language. As a complete and sophisticated language, mastering sign language is no simple task. Sign Language Recognition (SLR) aims to reduce communication barriers between the deaf and others by interpreting gestures in sign language videos. However, due to the scarcity of multi-view annotated datasets, most current models are only capable of recognising hand gestures from frontal views. Performance can significantly drops when processing non-frontal views due to issues like hand occlusion. To address this issue, we introduce view-conditioned diffusion models for the SLR task. Leveraging the strong zero-shot generalisation capability of large-scale diffusion models, we synthesise multiple side-view pose estimations to enhance the diversity of training data. Additionally, we integrate the RGB modality to enrich skeleton-based methods with a broader contextual perspective. Experiments on the MM-WLAuslan dataset demonstrate that our method significantly improves the robustness of SLR models when processing multi-view input. In the Cross-View Isolated Sign Language Recognition (CV-ISLR) Challenge, we achieved second place in the RGB setting.