An End-to-End GPT-4o Informed Pipeline for Manipulation Task Planning with Sim2Real Validation

Nowadays, the integration of task planning for robotic manipulation with LLMs is at the frontier of research, where LLMs act as critics or assistants, serving various purposes such as feasibility prediction, object pose estimation, motion trajectory planning, and obstacle avoidance. Despite the promising progress in this field, most of these works still focus on leveraging LLMs to enhance the performance of a single part of robotics task planning, while neglecting the integration of LLMs into the overall task planning process from scratch to finish. Thus resulting in potential risks to the system’s generalizability and adaptability in real-world experiments. To address the above challenges, we propose a novel end-to-end pipeline where GPT40 are globally integrated into the entire task planning process. This pipeline encompasses scene setup and description, task planning, relevant code generation, and sim-to-real validation, reducing labor costs while improving the efficiency of task planning, thereby demonstrating the feasibility and advantages of our pipeline. The demo video of our work can be seen here: https://youtu.be/u2rkymOjVQ4