An Innovative System for Real-Time Translation from American Sign Language (ASL) to Spoken English using a Large Language Model (LLM)

For millions of people with hearing difficulties, sign language of some form is used as the primary form of communication in many day-to-day situations. Considering the relative infrequency in sign language fluency globally, it is imperative that a novel system is created to facilitate communication for individuals with hearing impairments, especially in time-critical situations such as in medical environments. Ultimately, this research aims to create a system that is portable, intuitive, and easily usable in an individual’s day-to-day life. This research project explores the efficiency and practicality of using a Large Language Model (LLM) to translate phrases in American Sign Language (ASL) directly to English sentences in real-time. The proposed system utilizes the Leap Motion Controller as a method to capture signing information from the user, where this captured information is transformed into glosses (word representations of signs) and provided to the proposed LLM for translation. The experimental results for both accuracy of sign recognition and ASL to English sequence translation show that the proposed system is promising.