Semantic-Guided Gaussian Splatting with Deferred Rendering

Revolutionizing novel view synthesis, 3D Gaussian Splatting has unlocked new horizons in 3D visual representation. Despite the efficiency and impressive rendering capabilities of GS, the accurate inverse rendering of reflectant non-Lambertian surfaces remains a significant challenge, particularly in the context of diverse reflective materials settings, leading to inconsistent renderings and undermining the technology’s potential in applications ranging from digital asserts production to virtual reality. We propose Semantic-Guided Gaussian Splatting (SGGS), which aims to address this challenge by leveraging the capabilities of semantic features derived from cutting-edge 2D foundation models, revolutionizing material properties optimization for Gaussians. By integrating this high-level understanding, we enhance the model’s resilience against reflective surfaces and significantly improve multi-view consistency, which is a crucial step towards seamless immersive experiences. Our experiments systematically demonstrate that SGGS outperforms previous methods in terms of both rendering quality and geometry.