Unraveling temporally entangled multimodal interactions: investigating verbal and nonverbal contributions to collaborative construction of embodied math knowledge

Abstract In various technology-enhanced learning (TEL) environments, knowledge co-creation progresses through multimodal interactions that integrate verbal and nonverbal modalities, such as speech and gestures. This study investigated two distinct analytical approaches for analyzing multimodal interactions—triangulating and interleaving—by applying them to collaborative learning processes during an online embodied mathematics intervention. The findings demonstrate that the interleaving approach captures the temporal dynamics and nuanced interplay between multimodal events, providing deeper insights into how shared meaning-making evolves over time. In contrast, the triangulating approach effectively identifies cumulative interaction patterns but does not account for their temporal structure. Specifically, the interleaving approach, employing epistemic network analysis, revealed statistically significant differences in discourse patterns between learners with larger and smaller variances in upper body movements during the co-design activity. These findings underscore the complementary value of the interleaving approach in analyzing multimodal interactions and offer practical implications for advancing understanding of collaborative learning processes in TEL environments.