Transformer Model for Direct Translation of Spoken Text to Arabic Sign Language (ArSL)

Arabic Sign Language (ArSL) is a formal language used by Deaf and Hard of Hearing communities. It helps them understand Arabic spoken by hearing people, particularly in formal speech, such as news or sermons. The production of sign language systems can support communication by translating spoken languages into ArSL videos. However, current systems often fail to meet user needs due to poor translation of grammatical structures and limited ability to capture the textual meaning of speech. This research focused on a neural machine translation (NMT) approach to convert spoken Arabic texts into Arabic Sign Language (ArSL) using a transformer-based model to overcome these difficulties. The ArSL video of the sign is produced as a series of 2D skeletal poses. The proposed methodology is based on an encoder-decoder architecture: texts are represented by a transformer model, and a posture decoder generates gestures. The model is evaluated using quantitative criteria, such as mean square error (MSE) and the dynamic time warping (DTW) measure, as well as qualitative evaluations by sign language experts and users from the Deaf and Hard of Hearing community. The study uses Friday sermon data as a reference to build the translation model and applies optimization techniques such as temporal modulation, noise addition, and resizing to enhance performance. The results show that the enhanced model improves the quality of gesture representation, reducing DTW by 43.4% relative to the baseline in the non-continuous ArSL test. On the other hand, in the Continuous ArSL test using Short Sermon, the DTW is reduced by around 2.25%. In addition, the enhanced model received high user acceptance, with 75.2% satisfaction. Moreover, unified qualitative evaluation criteria for sign language production are proposed, including clarity, accuracy, consistency, and naturalness, which could serve as a basis for future evaluations in this field. These criteria are used to evaluate the generated ArSL and measure user satisfaction.