Flexible voice control for collaborative robotics

Abstract In this paper, we present a novel combination of an artificial intelligence (AI)-based and a grammar-based approach for a voice control system (VCS) for collaborative robots in industrial environments. A behavior tree (BT) serves as a high-level control system. A zero-shot object detection model is utilized to ensure the detection of different objects in the workspace. The grammar-based VCS is used for predefined actions. If a command does not trigger any action, a fine-tuned large-language model (LLM) is utilized for task planning. In order to further improve the model, interactions with the LLM are stored and processed to collect additional training data. The system is evaluated in real-world experiments, using a collaborative robotic arm in conjunction with a static calibrated depth camera that provides a birds-eye view.