LIVE-GS: LLM Powers Interactive VR by Enhancing Gaussian Splatting

We propose LIVE-GS, a highly realistic interactive Gaussian splatting system in VR environments powered by LLM. Our pipeline supports reconstructions and physically-based interactions in VR, integrating object-aware reconstruction, GPT-assisted inpainting, and a computationally efficient simulation framework. To enhance scene understanding, we prompt GPT-4o to analyze the physical properties of objects in the scene, thereby guiding physical simulations to align with real-world phenomena. Our experimental results demonstrate that with the assistance of LLM’s understanding and scene enhancement, our VR system can support complex and realistic interactions without requiring additional manual design or annotation.