Enablers of low-latency immersive interaction in future remote-rendered Mixed Reality applications

Mixed Reality (MR) has launched from science fiction but its entrance in reality can reshape our society. By combining the physical and virtual worlds, it provides novel ways of immersive interactions and experiences and by these means enables a new generation of applications. The most exciting and challenging ones support collaborative, multi-user operation in large geographical scale, require real-time environment comprehension and high visual fidelity. The success or failure is definitely impacted by the capabilities and performance limits of edge cloud platforms and 5G/6G networks providing the offloading features for CPU/GPU intensive MR functions. In addition, the desired quality of user experience calls for further mechanisms at the application level hiding the consequences of varying network characteristics. In this paper, we propose a novel edge cloud based architecture for future remote-rendered MR applications supporting low-latency immersive interactions. Our contribution is threefold. First, the system architecture is presented focusing on the remote rendering, 3D simulation and environment detection control loops. Second, we highlight the main features of our proof-of-concept prototype and our dedicated application, namely the Mixed Reality version of a Rocket League inspired game. Third, the concepts are validated via experiments in a Beyond 5G infrastructure where we analyze the operation and latency characteristics of the overall system. In addition, the quality of the user experience is also evaluated via real-life experiments conducted as part of a student competition. The results show that the latency and jitter characteristics of the most sensitive render loop can be managed efficiently together by a network-level control (slice priorities) and an application-level (dynamic jitter buffer) mechanism.