Orbis: Redesigning Neural-enhanced Video Streaming for Live Immersive Viewing

Emerging live immersive viewing systems require streaming large 360 videos to users via limited wireless bandwidth. Neural-enhanced video streaming offers a promising solution by streaming down-scaled videos and enhancing them by client computation. However, prior systems treated video downscaling and enhancement separately, overlaying existing enhancement techniques onto current video infrastructure to accommodate legacy downscaling methods. This supplemental client design has led to spatial information loss and prohibitive model overheads in 360 video streaming systems. This paper presents Orbis, a redesigned, holistic neural-enhanced video streaming framework that integrates complementary down-scaling and enhancement for live immersive viewing. Orbis is empowered by an enhancement-driven interleaved downscaling approach, an inpainting-based enhancement model tailored to interleaved data, and a multi-scale tile adaptation scheme that optimizes immersive viewing experience in dynamic environments. Experimental results show that Orbis improves viewing experience by up to 60% and reduces wireless bandwidth by up to 49% compared to the best-performing baseline.