Speech2Sign with ASL-Transformers and SL-GAN

Sign language is an essential means of communication for the deaf and hard-of-hearing, yet its translation and production are uniquely challenging due to its multimodality and the ambiguity of aligning spoken language sentences with visual signs. The objective of automatic Sign Language Production (SLP) is to transform speech into sign language videos that closely match the accuracy and expressiveness of a human interpreter, facilitating communication between the deaf community and the predominantly speaking world. Many prior SLP works have taken a gloss-free approach at the expense of accuracy, or have produced low-quality 2D sign poses as the final output, failing to convey the intricacies of sign language. Contrary to these approaches, we demonstrate the merits of gloss, a well-crafted loss function, and a multi-scale Generative Adversarial Network (GAN) in the production of accurate and photorealistic sign language videos. Our approach is a multistage SLP framework that begins with translating transcribed speech to its written sign language notation (gloss) using our ASL-Transformers model. We then map the gloss to its dictionary sign frames through a database query and generate the corresponding 2D pose sequence with DWPose, a pretrained pose estimator. Finally, we synthesize a photorealistic sign language video from the 2D pose sequence using our SL-GAN, a pose-conditioned image generation model. We evaluate our ASL-Transformers on the ASLG-PC12 dataset, reporting a BLEU-4 score of 89.58, and our SL-GAN on the ASLLVD dataset, reporting a mean SSIM value of 0.921.