Physics and geometry-augmented neural implicit surfaces for rigid bodies

This paper tackles the challenges of physics-based simulation of rigid bodies in neural rendering, with a focus on 3D model representation and collision handling. We propose Physics and Geometry-Augmented Neural Implicit Surfaces (PGA-NeuS), a novel approach that combines neural implicit surfaces with a differentiable physics solver. In the pre-processing stage, PGA-NeuS reconstructs static scene and object geometry from multi-view images using signed distance fields (SDFs). For dynamic scenes captured in monocular videos, these SDFs, along with the initial position and orientation of moving rigid bodies, are fed into a differentiable rigid body solver to optimize physical parameters, such as initial velocity and friction coefficients. Subsequently, PGA-NeuS leverages color loss, physics loss, and object mask supervision to iteratively refine the neural implicit surface, ensuring the target object's alignment with the predicted motion sequence. We evaluate PGA-NeuS on five real-world scenes, demonstrating its ability to accurately reconstruct realistic motion sequences and estimate physical parameters such as position and velocity. Dataset and source code are available at https://github.com/Raining00/PGA-NeuS . • PGA-NeuS reconstructs moving rigid objects from monocular videos using physics-aware neural surfaces. • Joint optimization of color, physics, and mask losses enables dynamic scene reconstruction from monocular videos. • We introduce a dataset with synthetic and real scenes featuring sliding, rolling, and collision motions.