3DS-Plan: A Perception-Planning-Co-Design Framework to Facilitate Robot Task Planning with Open-Vocabulary 3D Scene

To plan and execute robotics tasks, 3D scene perception and understanding are critical for robots to interact with complex environments effectively. Traditional systems often rely on closed vocabularies and are constrained by pre-defined object categories, which limit their flexibility. In this paper, we propose 3DS-Plan, an open-vocabulary 3D scene perception and generation model designed to facilitate long-horizon task planning. It leverages the pre-trained foundation models to support robotic scene understanding and further provides environmental details to infer actionable steps in various scenarios. We validate the effectiveness of our frame-work through comprehensive experiments. It demonstrates a 31.41 % improvement in computational efficiency for class operations and at least a 14.52 % increase in success rates for complex tasks without requiring extensive retraining or extra annotation. The corresponding project page is available at https://techpage.github.i0/open3dsp/.