Accessible American Sign Language Learning in Virtual Reality via Inverse Kinematics

Along with the rapid advancement of Virtual Reality (VR) and the metaverse, interest in this technology has surged among game developers and in fields such as education and healthcare. VR has enabled the rise in immersive, gamified activities, whether for rehabilitation, therapy, or learning. Additionally, VR and Motion Capture (MoCap) have allowed developers to create further accessibility features for end-users with special needs. However, the excitement of using new technology often does not align with the end user’s use cases. The over-reliance on cutting-edge hardware can negatively impact most end users who lack access to such expensive tools. To this end, we conducted an inclusivity-focused study that enables learners to practice ASL in an immersive and engaging way using only head- and controller-based tracking. Our approach replaces full-body MoCap with Inverse Kinematics (IK) and simple controller mappings for upper-body pose and hand-gesture recognition, providing a low-cost, reproducible alternative to costly setups.