Empilo: Realizing Immersive Mobile 3D Video Conferencing through Parameterized Communication

In this work, we explore a new communication paradigm for immersive 3D video conferencing, termed parameterized communication, which dramatically reduces bandwidth usage by eliminating the need to exchange excessive volumetric data. Instead, this approach extracts a compact set of informative parameters representing key elements in the 3D space, transmits only these parameters, and reconstructs the scene on the receiving end. Translating this concept into practice, we present Empilo, a mobile 3D conferencing system composed of a face parameter extractor and a neural rendering-based scene generator. However, while neural rendering excels at synthesizing arbitrary views of objects without explicit 3D models, its heavy computational demands present a major obstacle for mobile deployment. To overcome this challenge, we propose a novel technique called truncated ray marching, which significantly reduces computational overhead by replacing iterative MLP inferences with a single-pass of a shallow neural network. Furthermore, to ensure a consistently immersive experience, we structure the neural-free lightweight renderer as a decoupled component, dedicated to delivering rapid responsiveness to dynamic viewpoint changes. These breakthroughs on computation together enable Empilo to rely entirely on mobile resources, achieving real-time performance with a frame generation time of 30.3 ms and a re-rendering latency of just 6.6 ms—all while operating at an exceptionally low bitrate of 24 kbps. Our approach provides valuable guidance for the practical deployment of 3D conferencing, envisioning accessibility on par with platforms like FaceTime and Zoom.