Action Contextualization: Adaptive Task Planning and Action Tuning Using Large Language Models

Large Language Models (LLMs) present a promising frontier in robotic task planning by leveraging extensive human knowledge. Nevertheless, the current literature often overlooks the critical aspects of robots' adaptability and error correction. This work aims to overcome this limitation by enabling robots to modify their motions and select the most suitable task plans based on the context. We introduce a novel framework to achieve <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">action contextualization</i>, aimed at tailoring robot actions to the context of specific tasks, thereby enhancing adaptability through applying LLM-derived contextual insights. Our framework integrates motion metrics that evaluate robot performances for each motion to resolve redundancy in planning. Moreover, it supports online feedback between the robot and the LLM, enabling immediate modifications to the task plans and corrections of errors. An overall success rate of 81.25% has been achieved through extensive experimental validation. Finally, when integrated with dynamical system (DS)-based robot controllers, the robotic arm-hand system demonstrates its proficiency in autonomously executing LLM-generated motion plans for sequential table-clearing tasks, correcting errors without human intervention, and showcasing robustness against external disturbances. Our framework also features the potential to be integrated with modular control approaches, significantly enhancing robots' adaptability and autonomy in performing sequential tasks in the real world.