Large language model-based copilot for planning and executing of co-active USAR missions using actionable semantic scene graphs

Remote operation of robots in urban search and rescue (USAR) scenarios can impose significant physical and cognitive demands on human operators. To alleviate this, modern robotic systems can offer (semi-)autonomous capabilities and teleoperation assistance. However, effective mission planning remains difficult due to the complexity and variability of disaster environments. In this work, we introduce a novel AI assistant system designed to support the planning and execution of such USAR missions by accepting natural language task descriptions and generating grounded task plans for the robotic system. Our copilot system facilitates coactive planning, dynamically adapting between autonomous execution, assisted operation, or teleoperation based on expected costs and operator-provided specifications. We combine the capabilities of large language models (LLMs) and graph optimization, leveraging information from a scene graph-based environment representation to propose a novel, robust framework that translates natural language instructions into a grounded sequence of actions. Our evaluations in a complex USAR scenario show superior and more robust planning performance over direct LLMs prompting, especially for long-horizon tasks and in reliably avoiding task-related hallucinations.