Students’ experience and learning outcomes in multisensory environments: the moderating role of interaction modalities

Abstract Smart learning environments (SLEs) utilize technological advancements to facilitate effective, engaging, and personalized learning experiences. They depend on sensors and advanced connectivity to gather information and make informed decisions. Multisensory Environments (MSEs) naturally align with and enhance the capabilities of SLEs offering new opportunities to enhance learning effectively, and engage children with stimulating educational experiences leveraging different interaction modalities. Investigating how children interact with these new systems is important to design educational technologies. However, limited research has been conducted to evaluate the role of interaction modalities in moderating the relationship between students’ experience and their learning outcomes in a MSE. We, therefore, tracked 175 students’ (aged 6-10) correctness rate to questions and their states through motion, heart rate, and electrodermal activity, obtaining their levels of fatigue, stress, engagement, emotional regulation, and anxiety. We then analysed the moderating role of five different interaction modalities ("card", "feet", "hands", "voice", "wand") on the relationship between correctness rate and states. The results of this in-situ study show that the relationship between student states and their performance is moderated by the interaction modalities, offering important design and theoretical implications on the role of the interaction modalities in the learning experience of students with an MSE. The contributions of this research benefit all stakeholders involved, including students who receive appropriate learning experiences, and practitioners who can make informed decisions on what interaction modalities to use to support the learning experience.