Embodied Assistant: Robot Mobility Operations Guided by Open Vocabulary in Open Environments Utilizing LLM

In the field of artificial intelligence and robotics, enabling robots to understand and execute complex tasks in unknown and open environments through natural language has become a frontier of current research. Traditional methods that rely on predefined action libraries face significant limitations when dealing with environmental diversity and task uncertainty. Accordingly, this study introduces a novel embodied intelligence framework named ‘Embodied Assistant’, which utilizes large language models (LLMs) for open-ended reasoning and adaptive task planning, can autonomously guide robots to flexibly complete challenging tasks in complex scenarios without relying on fixed action templates. By establishing effective multimodal LLM interaction pathways, this framework grants robots the ability to perceive and understand the task environment, concurrently allowing them to accurately interpret natural language instructions, and to independently plan action strategies along with robust robot trajectories. Moreover, the introduced multi-level task feedback mechanism effectively enhances the robots’ ability to self-correct and replan when they encounter planning failures. In extensive real-life testing scenarios, the proposed method achieved an 83.3% task success rate across various combined mobility and grasping tasks, significantly outperforming the most advanced baseline methods. A detailed analysis of robot fault recovery further demonstrates the substantial potential of this method in practical applications. Prompts and videos are provided at the project homepage: embodied-assistant.github.io