System As A Collaborator (SAAC): a Framework for Modeling, Capturing and Augmenting Collaborative Activities in Extended Reality

When humans collaborate on a shared task, they use a myriad of verbal, para-verbal and non-verbal cues to achieve this end. Modern Artificial Intelligence sensing techniques such as Social Signals Processing (SSP) allow the characterization of users’ activities in Augmented or Virtual Environments through the analysis of heterogeneous multimodal data sources, such as interaction actions, gaze direction, avatars’ positions, and speech analysis. Therefore, this enables realtime assessment of team processes, including communication or team situation awareness. In return, it can provide context-specific feedback and information to augment team capabilities. However, implementing this vision remains a technical challenge, particularly in gathering real-time data sources in multi-user collaborative scenarios. This paper presents a framework for assessing and augmenting team collaboration with Extended Reality Environments (XRE). We propose a multimodal architecture to capture users’ activities and augment the team capabilities by displaying system-generated context-specific collaborative cues. Our proposed vision, System As A Collaborator (SAAC), frames the XRE as a direct actor embedded in the collaborative activity, improving team members’ capabilities with collaborative augmentations instead of being merely the space where collaboration occurs. We demonstrate the feasibility of our approach and system architecture through use cases of experimental XRE where our software architecture collects multimodal, heterogeneous, and multi-user interaction, behavioral and physiological data, and generates direct, reactive cues, and higher-level context-specific cues, augmenting team collaboration.