A Case of Cups and a Ball: Utilizing Generative Artificial Intelligence for Human-robotic Collaboration in Task Execution

Within the domain of robotics, effective interaction between a human collaborator and robot is a topic of great importance. With the continued advancements in Generative Artificial Intelligence (GAI), we propose a prototype system that utilizes GAI and seeks to bridge the gap between the human and robot. This prototype utilizes a Large Language Model (LLM) combined with a vision system for detection and tracking, to allow the GAI system to control a robotic arm and manipulate objects within the action space. We model the action space within our experiments on the 3-cup Monte game where the user can prompt the system to manipulate the cups within the action space based on the ball’s location. Experimentation is done on prompts of varying lengths of sequences, and we evaluate our system based on its ability to understand the prompt and correctly execute the desired task. With this collaboration of techniques and technologies, we hope to open the possibilities and extend human and robotic collaboration into new areas. The results of our experiments show promising results to this end.