Web SMILE demo: a web application providing automated feedback on sign language vocabulary production

In language learning, learners need to develop competence in comprehension and production, both of which are necessary for successful interaction. The use of digital technologies to aid the acquisition of such competencies has proven effective in spoken language learning and is emerging in sign language learning. Most existing sign language learning tools are designed for comprehension acquisition, while the production side involves only self-comparison. However, for learning effective sign language production, good proprioception, spatial reasoning, and observation skills are necessary. There is a need to develop applications that guide learners in various aspects of sign language production. For this reason, we developed an artificial intelligence (AI)-driven, web-based sign language learning application in which sign language production is assessed automatically using neural networks and hidden Markov models. The application guides the learners by providing video feedback and scores at different levels, namely, at (i) the sign level, (ii) the form level (e.g., handshape correctness), and (iii) the spatiotemporal level. This paper describes the development process of the application, including its architecture, interface, and AI methods. It also covers the intermediate stages, including evaluation studies of both the user interface and the underlying AI model. These studies were conducted with sign language learners and human raters to validate the AI methods underlying the application and feedback designs, and to demonstrate the feasibility of the proposed system. The present study underscores the challenges and key elements to consider when designing such evaluation tools. These elements include the choice of sign language assessment algorithms, the importance of linguistically valid data, the design of effective feedback, the importance of incorporating learners’ perspectives, and utilizing human ratings to validate the tool.