Turkish Sign Language Video Generation from Pose Sequences Using a Conditional GAN Model

Sign language is a visual gestural language that uses hand shapes, facial expressions, and body movements to convey meaning, and it is essential for communication among individuals with hearing impairments. Sign language users often face challenges in environments where sign language is not readily available. That is primarily because producing sign language requires a real signer, which can be time-consuming and labor-intensive. One potential solution to make more available sign language to its users is the use of artificial intelligence to generate sign language. While there have been global studies exploring AI-generated sign language videos for various sign languages, the research in this field is still evolving and active. Specifically, for Turkish Sign Language (TSL), only a few studies have been conducted. Our study introduces a model that combines pose sequences with reference images of signers to generate authentic sign language videos. Utilizing a conditional generative adver-sarial network (cGAN), the model generates video sequences by particularly focusing on complex finger details, arm positions while preserving the appearance of the reference image. The AUTSL dataset is employed for training and testing our model. Our early results demonstrate our model's capability to produce realistic hand and arm movements while effectively capturing finger details; we got 45.5 with Th e Frechet inception distance (FID), 35.8 with the peak-signal-to-noise ratio (PSNR), and 0.9 with the structural similarity index measure (SSIM).