Multi-robot task planning for multi-object retrieval tasks with distributed on-site knowledge via large language models

Abstract It is crucial to efficiently execute instructions such as “Find an apple and a banana.” or “Get ready for a field trip,” which require searching for multiple objects or understanding context-dependent commands. This study addresses the challenging problem of determining which robot should be assigned to which part of a task when each robot possesses different situational on-site knowledge, specifically, room-wise object presence probabilities learned from the area designated to it by the user. We infer room-wise object presence probabilities via Bayesian inference using a spatial concept model. The inference results are then converted into prompts. Large language models (LLMs) use these prompts to decompose instructions into tasks and assign them to multiple robots. We designed a novel few-shot prompting strategy that enables LLMs to infer required objects from ambiguous commands and decompose them into appropriate subtasks. In our experiments, the proposed method achieved 47/50 successful assignments, outperforming random (28/50) and commonsense-based assignment (26/50). Furthermore, we conducted qualitative evaluations using two actual mobile manipulators. The results demonstrated that our framework could handle instructions, including underspecified instructions such as “Get ready for a field trip.”, by successfully performing task decomposition, assignment, sequential planning, and execution. For reproducibility, we release the full set of prompts on the project website at https://kentomurata0610.github.io/multi-robot-task-planning .