Deep Reinforcement Learning for Next Best View Planning in Autonomous Robot-Based 3D Reconstruction

This paper presents an enhanced approach to automate the robot-based Next Best View planning in 3D reconstruction using Deep Reinforcement Learning (DRL). In our study, a robot-guided sensor captures the object from multiple viewpoints, with each viewpoint autonomously selected by a DRL agent to optimize scan efficiency and reconstruction quality. Our reward function accounts for both robot’s movement and scan quality, enabling the agent to shorten its path while maintaining high scan quality. Unlike other methods, we utilize a continuous action space, offering greater flexibility than approaches with predefined action sets or regions of interest. Additionally, the agent autonomously determines when to stop the scanning process, further enhancing performance. The agent is trained and tested in a simulation environment using the state-of-the-art ABC dataset, and is later evaluated on new, unseen parts, where it consistently outperforms the selected simplistic baseline method. The system is designed for easy extensibility, allowing for the integration of additional optimization criteria, new datasets, and further training.