Prompting LLM for Embodied Tasks with Expert Instruction and Dimension Separation

With the development of large language models, it has become possible to use multimodal models to build embodied agents. Currently, most approaches for handling embodied intelligence tasks with multimodal large models rely on prompt engineering or fine-tuning the models. However, to construct a general embodied intelligence model, the model needs to transfer world knowledge to corresponding embodied intelligence scenarios and spatial contexts. Language models that have not been optimized for embodied intelligence scenarios lack a good understanding of 3D scenes and the robot's action space. We attempted to address this issue in two ways, separated prompting approach where the model decomposes 3D spatial problems into 2D spatial problems and a trained expert model to guide the general model in spatial perception and action execution. Ultimately, the general model achieved improvements in embodied intelligence tasks. Experimental results demonstrated the effectiveness of our method.