Encoding, decoding, and rendering pipeline for multiview streaming and display in VR using VVC

Virtual reality (VR) head-mounted displays (HMDs) require high bitrates for immersive, high-quality visuals. Versatile Video Coding (VVC) significantly improves compression efficiency, making it ideal for VR multiview streaming and display. However, VR multiview streaming and display face two challenges: ultra-high bandwidth and ultra-low latency requirements for an optimal user experience. This paper presents an integrated, end-to-end pipeline to address these challenges, including (i) field of view (FOV)-based bitrate allocation, (ii) adaptive streaming for dynamic network conditions, and (iii) real-time decoding and rendering optimizations. We explore FOV-based bitrate allocation to prioritize user’s focus area and adaptive streaming algorithms to handle network fluctuations. For high-end devices like Vision Pro, we implement a software-based decoding and rendering approach with VVC, minimizing latency and improving rendering efficiency. This work presents a VVC-based pipeline for multiview streaming and display in VR, focusing on bitrate allocation, adaptive streaming, and real-time decoding and rendering optimizations.