ChainBot: An Agent System for Autonomous Robotic Object Manipulation by Dynamically Chaining Multiple Foundation Models

This paper proposes ChainBot, an agent system that autonomously plans and executes robotic object manipulation tasks based on a user's natural language instructions by dynamically chaining multiple foundation models. The ChainBot system uses a large language model (LLM) for effective task planning. To ensure the generation of correct task plans, the system enriches the LLM prompt with useful context, including plan examples and domain-specific heuristic knowledge. Before execution, ChainBot also uses a secondary LLM for pre-execution validation. If an error is detected, the system replans the task using the validation result as feedback. During task execution, ChainBot employs a large multimodal model (LMM) to monitor the process for quick error discovery and recovery. This is achieved by checking both the preconditions and postconditions of each action using real-time camera images captured before and after execution. As a result, ChainBot achieves fully autonomous robotic manipulation without user intervention when implemented as an agent system using the LangChain and LangGraph frameworks. Through various quantitative and qualitative experiments, this paper demonstrates the superiority of ChainBot compared to conventional systems.