Insights from Immersive Learning: Using Sentiment Analysis and Real-time Narration to Refine ASL Instruction in Virtual Reality

Immersive virtual reality presents a rich opportunity for learning signed languages, given the immersive environment’s ability to represent three-dimensional information. We developed a proof-of-concept American Sign Language (ASL) learning in immersive virtual reality (VR), named ASL Champ! Twelve hearing non- or novice signers played one full level of the game, during which they were asked to provide concurrent think-aloud (CTA) commentary, narrating their experience as they played in real time. We conducted a sentiment analysis from recordings of the CTA and subsequent open-ended questions and qualitatively assessed the narrations for salient themes. The analysis revealed specific aspects of the users’ experiences that were most likely to lead to positive or negative expressions during the CTA and the question session. The factors that had the most impact on user sentiment were the success of the sign recognition in the game and the extent to which users found the game intuitive or self-explanatory. We also found that users with more technology anxiety were more positive about the game. We also qualitatively examined user comments, revealing their real-time game experiences. This work provides insights into which aspects of an ASL learning VR game are most important for user experiences. We conclude with takeaway recommendations for future virtual or augmented reality sign language learning games.