LLM-driven agent for speech-enabled control of industrial robots: A case study in snow-crab quality inspection

This study investigates the integration of large language models (LLMs) into a voice- and vision-based robotic control system in autonomous industrial applications. The main objective is to demonstrate that an LLM-based agent can interpret natural instructions, dynamically plan movements, and execute robotic actions without domain-specific supervised learning, thereby enabling autonomous robotic planning. The proposed system relies on a voice interface, an LLM agent, and tools for real-time robot control. A dedicated communication module was developed to ensure the full control of a KUKA industrial robot using WebSocket, without resorting to proprietary solutions. To validate the approach, a case study was conducted using a robotic cell, which was applied to snow-crab sorting, where computer vision provides real-time perception. The experimental evaluation covered a wide range of commands, including movement instructions, complex planning tasks (e.g., trajectory generation), and visual queries based on crab quality, size, and anatomy. The results showed that the model exhibited robust interpretation capabilities with an overall success rate of 98.46%. These performances highlight the potential of LLMs to facilitate human-robot interaction in real industrial environments, while reducing programming complexity and increasing system autonomy. • A speech-enabled system controls an industrial KUKA robot using a large language model. • Python tools enable real-time motion planning and robotic execution. • Visual inspection of snow crabs guides automated sorting tasks. • Achieves 98.46% overall success rate across complex and noisy commands. • Demonstrates real-world deployment in an industrial robotic cell.