SeeMusic: XR-Enhanced Musical Language Training Application for Hearing-Impaired Children

Augmented Reality (AR) and Virtual Reality (VR) are technologies that achieve auditory compensation by providing visualization of auditory information, making them increasingly used to enhance the language learning experience of hearing-impaired children. However, a limitation of current applications is the failure to follow hearing recovery principles in designing VR and AR experiences, resulting in a lack of specificity. This paper proposes an innovative mobile application, SeeMusic, which integrates music therapy methods with AR and VR visualization techniques based on a listening skill hierarchy to enhance auditory rehabilitation and language learning for hearing-impaired children. By applying Erber's Hierarchy as hearing recovery principles, this method provides a self-directed learning process, enabling children to practice auditory skills and develop speech production through an AR textbook, singing practices, and a VR game. Usability was assessed through teacher evaluation, with an average score of 4.67 to 5.00 across most categories. The WIPI test with hearing-impaired children showed a significant improvement, with an average increase of 18 points.