University Students' Perceptions of a Multimodal <scp>AI</scp> System for Real‐World Collaboration Analytics: Lessons Learned From a Case Study

ABSTRACT Background Many researchers work on the design and development of multimodal collaboration support systems with AI, yet very few of these systems are mature enough to provide actionable feedback to students in real‐world settings. Therefore, a notable gap exists in the literature regarding students' perceptions of such systems and the feedback they generate. Objectives This study designed, built and implemented a set of collaboration analytics to capture, interpret and provide feedback on students' collaborative processes, including their non‐verbal group interactions as well as group challenges and regulation arising from discourse in authentic collocated collaborative settings. Methods Seven groups of five to six postgraduate students with varying backgrounds participated in face‐to‐face collaborative design tasks ( n = 36) for an 11‐week‐long semester. Multimodal data from audio and video recordings of collaborative learning sessions were analysed using various machine learning techniques to model students' group processes and to generate feedback. A post hoc evaluation of the collaboration analytics feedback was conducted using individual student reflections and focus group interviews. Results and Conclusions The findings suggest that analytics feedback has the potential to promote students' understanding of their collaborative processes (e.g., awareness of individual, peer and group behaviours and alterations at the individual level). However, the study also identified significant limitations and challenges associated with the real‐world application of collaboration analytics (e.g., limited group transactions stemmed from a lack of group interpretative sessions). The paper concludes with a discussion on future design suggestions and principles (e.g., an integration of analytics with the learning design, value alignments among stakeholders and roles of teachers).