Simulation-Based Learning in Virtual Reality: Three Use Cases from Social Science and Technological Foundations in Terms of Va.Si.Li-Lab

Abstract This article examines the predictability of communication scenarios within the context of simulation-based learning in virtual reality (VR). The aim is to investigate multimodal patterns of social interaction that accompany human communication in conflict situations. Understanding these patterns can ultimately enhance educational technologies’ ability to address problematic learning situations and support learners in benefiting from VR-based learning. To achieve this, the system must accurately predict the task context. A central goal of this article is to shed light on this potential. Additionally, our research extends to visual communication beyond purely linguistic interactions, aiming to enhance VR immersion in communicative practices. To this end, the article examines the associations between multimodal information units generated by individuals interacting in three distinct learning scenarios: work organization, school pedagogy, and social life. Several experiments demonstrate that predictability exists when multimodal communication is analyzed at the level of eight coarse-grained modalities, including speech, head and body movements, and gestures. The interactions are observed in VR using Va.Si.Li-Lab , a simulation-based system that virtualizes learning scenarios, enabling participants to collaboratively manage potentially conflicting tasks through multimodal communication (Mehler et al. in: Duffy (ed) Digital human modeling and applications in health, safety, ergonomics and risk management, Springer Nature Switzerland, Cham, 2023). The article discusses the technology underlying Va.Si.Li-Lab , its database, and the post-processing of interaction data, including speech data. It provides theoretical motivation for the application scenarios and presents experimental data to illustrate the system’s usefulness. Based on these data, the article details experiments on the multimodal detection of social scenarios, positioning Va.Si.Li-Lab as a use case in simulation-based learning.