Clip, Copy, and Share: On-the-Fly Gaussian Splatting-Based Virtual Replicas for Remote XR Collaboration

Effective remote collaboration in eXtended Reality (XR) requires realistic and interactive 3D spaces, but existing systems are often hindered by complex setups, such as specialized sensors or pre-made 3D models, limiting their accessibility. To challenge these limitations, we propose a novel remote XR collaboration system that leverages 3D Gaussian Splatting (GS) to reconstruct high-fidelity environments on-the-fly during collaboration, using only a standard RGB camera on the commercial XR device. The core of our system is an interaction technique we term ’Interactive Gaussian Clipping,’ which enables users to instantly clip manipulable, photorealistic ’Gaussian Replicas’ from any part of the newly generated scene. Then, the Gaussian Replicas are copied and shared between remote collaborators instantly. This approach eliminates the need for prior 3D modeling and specialized hardware, significantly lowering the barrier to remote collaboration. We validated the proposed approach through a within-subject user study (N=28) that compared the effectiveness of our Gaussian Replica (GR) method against a conventional 3D sketching (3DS) baseline in a collaborative assembly task. The results show that the GR method led to significantly faster task completion, a lower cognitive load, and a higher usability for local users. Furthermore, a majority of the participants (93%) preferred the GR method. Our work demonstrates a practical and accessible direction towards more effective and immersive remote collaboration, highlighting the potential of user-driven GS interaction in HCI.