Sign Language Transformer using Spatial Representations

Sign Language recognition has been studied and designed by many currently existing models but only a few models exist that focus on translation. We believe translation is very much needed for efficient communication between the disabled and abled. The proposed model not only does the recognition but also translates the sign language to understandable spoken language. The proposed project implements an end-to-end sign translation system that is capable of simultaneously learning both signs to gloss and gloss to text during the training process. This is done using transformers which consist of encoders and decoders, the encoder is used for recognition of sign language called gloss. Using an encoder transformer, the system understands the language by using the spatial-temporal nature of the language. Spatial understandings from the encoder are then sent through the decoder for translation tasks. The resultant of the decoder is the grammatically accurate sign language conversions We use PHOENIX-2014T dataset which consist of continuous sign videos of weather news reported in Germany with gloss and text. The proposed model is capable of sign to gloss, sign to text and sign to gloss to text.