Drink Serving Robotic Arm Relying on Multimodal Inputs

This paper introduces a voice-activated robotic arm with four degrees of freedom (DOF), developed to automate drink serving in a bar environment. The system incorporates an overhead camera and utilises the You Only Look Once (YOLO) architecture to detect and localise cups, obstacles, and the drink dispenser within the robot's operational workspace. Voice commands are processed using a Squeezeformer-based speech recognition model, allowing customers to place orders verbally. Precise arm movements are achieved through Model Predictive Control (MPC), while cycloidal drives are employed in each joint to enhance accuracy and minimise backlash. Inverse kinematics are resolved via the Gradient Descent method. The object detection system achieved a mean Average Precision (mAP50) of 0.981, while the speech recognition model recorded a Word Error Rate (WER) of 0.081 on the test set. System performance was validated through a series of user trials, with the average time to complete a drink order recorded at approximately 30 seconds.