Think Before Execute: Embodied Reasoning of Supportive Tools for Robot Service with Large Language Models

The utilization of appropriate tools can significantly streamline task complexity and expand the boundaries of robot capabilities, indicating a high level of intelligence. Finding a useful tool is a non-trivial problem when the robot confronts complex tasks lacking detailed execution instructions. In this paper, we investigate the problem of robotic task execution in response to human verbal requests. Motivated by the development of Large language models (LLMs), we propose a robotic embodied reasoning framework conditioned on LLMs. It enables robots to rely on their physical manipulation system or exploit supportive tools in the scene to effectively complete their tasks. To this end, the environment knowledge about objects is modeled using a scene graph. More specifically, by utilizing the on-site knowledge in the scene graph and the general knowledge in LLMs, we develop an embodied evaluation module to conduct task executable analysis considering the robot's abilities and the associated objects in the environment. Then a series of execution sequences are generated to drive the robot to fruitfully finish the task. Extensive experiments conducted in various real-world scenes demonstrate the effectiveness and generalization of our proposed framework.