Innovating Robotic Garment Handling Through the Integration of Large Language Models and Behavior Trees

Natural language offers a highly flexible and intuitive way for humans to communicate tasks to robots. This paper explores the potential of Large Language Models (LLMs) agents in enhancing industrial robotics, with a particular focus on textile manufacturing and garment handling tasks. The primary goal is to reduce the level of expertise required to develop robotic applications. Our framework comprises three key components: an LLM for code generation and common-sense reasoning, a vision-language model for open-vocabulary visual recognition, and a specialized 3D object recognition model. We introduce a novel four-stage prompt engineering process that customizes prompts to meet the specific needs of behavior tree generation. This process facilitates the development of programs for complex tasks, including safety management and parallel processing in a multi-arm robotic workspace, ensuring the robot executes actions safely and reliably. A RealSense RGB-D camera provides depth information, combined with foreground-background segmentation and curvature analysis, allows for precise grasping and manipulation. Both real-world tests and simulated trials in a Gazebo environment underscore the potential of this approach, demonstrating the LLM's capability to enhance safe robotic operations in industrial settings.