LiveVV: Human-Centered Live Volumetric Video Streaming System

Volumetric video (VV) has emerged as a prominent medium within the realm of extended reality (XR) with advancements in computer graphics and depth capture hardware. Users can fully immersive themselves in VV with the ability to switch their viewport in six degree of freedom (DOF), including three rotational dimensions (yaw, pitch, and roll) and three translational dimensions (X, Y, and Z). Different from traditional 2-D videos that are composed of pixel matrices, VVs employ point clouds, meshes, or voxels to represent a volumetric scene, resulting in significantly larger data sizes. While previous works have successfully achieved VV streaming in video-on-demand scenarios, the live streaming of VV remains an unresolved challenge due to the limited network bandwidth and stringent latency constraints. In this article, we propose <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">LiveVV</monospace>, a holistic live VV streaming system that integrates multiview capture, scene segmentation and reuse, adaptive transmission, and real-time rendering. <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">LiveVV</monospace> features lightweight VV capture modules for easy deployment, processes static and dynamic content separately to reduce bandwidth consumption, and incorporates a VV adaptive bitrate streaming algorithm (VABR) to ensure fluent playback with high-quality experience. Real-world implementation and evaluation demonstrate that <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">LiveVV</monospace> achieves live VV streaming at 24 FPS frame rate with less than 350-ms latency on average, meeting the requirements of real-life application.