A multimodal representation framework for collaborative knowledge-building in an immersive astronomy simulation: using transmodal ordered network analysis

Abstract The complex processes of collaborative knowledge construction require a multimodal approach to capture the interplay between learners, tools, and the environment. While existing studies have recognized the importance of considering multiple modalities, there remains a need for a comprehensive framework that explicitly models the dynamics of knowledge representation and construction. Drawing on theoretical perspectives from collaborative knowledge-building, distributed cognition, and multiple representations in science education, we propose a multimodal representation framework that captures the diverse ways in which learners externalize, negotiate, and advance their understanding. We employ Transmodal Ordered Network Analysis to examine the interplay between knowledge representations across three distinct yet interconnected spaces: the virtual space of the digital environment, the conceptual space of internal knowledge, and the physical space of gestures. This approach enables a more granular and accurate modeling of the temporal dynamics and influences associated with different modalities. Investigating 16 groups of college students (n = 77) who utilized an immersive astronomy simulation in their introductory astronomy course, results reveal distinct patterns between high- and low-learning groups. Notably, high-learning groups demonstrated more frequent and stronger cross-modal connections, linking verbal explanations with digital representations within the simulation and with embodied representations through gestures. It extends the theory of multiple representations by demonstrating its importance not only for individual learning but also for collaborative processes. The findings highlight the need for designing learning environments and analytic approaches that can support and capture the rich multimodal interactions through which students co-construct scientific understanding.