A Mobile Communication System Based on Visual-Auditory Translation Using Turkish Sign Language

This study introduces a vision-based mobile communication framework engineered to facilitate real-time, bidirectional interaction between deaf or hard-of-hearing and hearing individuals. To mitigate existing communication barriers, the proposed architecture seamlessly integrates deep learning algorithms with Automatic Speech Recognition (ASR) within a unified mobile environment. For the sign-to-text translation module, a Gated Recurrent Unit (GRU) network is utilized to recognize and classify 226 distinct Turkish Sign Language (TSL) words. The temporal model, trained and evaluated using the RGB videos of the AUTSL dataset, processes MediaPipe-based spatial features extracted from video frames and achieves a test accuracy of 76.77%. Predicted signs are subsequently rendered as instantaneous textual subtitles for both participants. Simultaneously, the hearing user’s vocal input is transcribed via a Speech-to-Text (STT) mechanism and displayed in real-time on both users’ screens. By prioritizing a computationally lightweight design and evaluating application-level latency, the developed system demonstrates the feasibility of responsive mobile-based interaction. Consequently, this approach offers a highly scalable solution to substantially enhance accessibility across social, educational, and professional domains.