HyVDAR: A semi-supervised framework for high-fidelity 3D object reconstruction from sparse multi-view point clouds in virtual reality applications

In virtual reality (VR) applications, high fidelity 3D meshes are needed for immersive experiences, and this calls for high fidelity reconstruction of 3D objects from sparse multi-view point clouds. There are several approaches, like PointMVSNet and NeRF, which require dense supervision or are computationally expensive and are not very suitable for the use of semi-supervised frameworks with limited labeled data. To address the above gaps, in this research study, a novel semi-supervised approach, Hybrid View-Consistent Diffusion-Augmented Reconstruction (HyVDAR), is proposed for the reconstruction of 3D meshes for VR from sparse multi-viewpoint clouds with limited labeled data while ensuring the highest geometric fidelity and rendering efficiency. HyVDAR has three novel components to the above pipeline: (1) self-supervised cross-view geometric consistency trained with transformers is used for alignment, denoted as View-Consistent Feature Propagation (VCFP); (2) conditional denoising diffusion models are used for patching the missing parts in sparse point clouds, denoted as Diffusion-Augmented Point Cloud Completion (APC); (3) graph neural networks with pseudo-labeling are used for refining the coarse meshes with both labeled and unlabeled data through semi-supervised learning, denoted as Semi-supervised Mesh Refinement (SMR). Trained on a combination of ShapeNet (labeled) and Pix3D (unlabeled) dataset, HyVDAR improves over the best competitive model (PointMVSNet) on sparse inputs by decreasing Chamfer Distance (CD) from 0.013 to 0.010 (25% improvement), decreases the labeled data requirements by 60% compared to supervised baseline models and achieves real-time VR rendering at 65 FPS. HyVDAR shows a good reconstruction quality and computational efficiency compared to the benchmark methods, and it can be used as a general-purpose tool for high-fidelity 3D modeling in virtual reality, which will be beneficial for gaming, digital avatar and immersive experience content creation.