Multi-Sight Passthrough: A Real-Time View Synthesis with Improved Visual Quality for VR

Virtual reality (VR) headsets create stereoscopic images through near-eye displays, offering immersive visuals while isolating users from their surroundings. In this paper, we focus on improving the interaction between users and the real-world environment while wearing VR headsets. A Multi-Sight Passthrough (MSP) system is proposed to achieve high-quality near-eye views in real-time by sensor fusion. In the data acquisition section, we combine the information captured by two RGB cameras and four grayscale cameras. In the algorithm section, we propose a new view synthesis method based on Multiplane Images (MPI) where we design a Swin Transformer backbone to extract multi-scale features and fuse them with the deconvolution layer, enabling better utilization of feature information. Through simulations and experiments, we demonstrate that our method not only meets the requirements for real-time rendering but also delivers superior image quality compared to state-of-the-art methods, outperforming our baseline by 5.39 dB in PSNR and 0.04 in SSIM. In a cross-dataset validation experiment on the RealEstate10K dataset, our method also achieved an average PSNR of 32.15 dB and an SSIM of 0.96.