Multimodal Robotic Surgical Instrument Transfer and Sorting Platform: Scrub Nurse Robot

A scrub nurse is an essential member of a surgical team. An important duty of a scrub nurse is to ‘deal with surgical instruments’, meaning transfer in the intra-operative phase, and sorting in the post-operative phase. Nevertheless, these two tasks are labor-intensive and time-consuming, which may lead to human error due to fatigue. Moreover, continuous manipulation of sharp instruments may increase the chance of injury for scrub nurses. As a result, a robotic platform can be a better alternative. This work proposes a deep learning-based multimodal robotic framework that can work with surgeons with speech and image inputs. A new minimally invasive surgery (MIS) instrument dataset is built with orientated bounding boxes and occlusion labels. On top of it, a two-step model, which is powered by YOLOv7-OBB (with 95.6% precision) and occlusion classification model (92% accuracy), provides the position, orientation, instrument class, and occlusion information of each instrument. Together with an automatic speech recognition module and a robotic arm, physical validation is conducted to substantiate that the proposed framework is capable of assisting surgeons and human scrub nurses in grasping-related tasks by the integration of image and voice.