Body Pose Estimation and Hand Position Detection for Hand-by-Hand Human-Robot Interaction

Robot-human interaction is rapidly increasing in work environments and patient/elderly care centers. Human-robot collaboration is becoming increasingly prevalent in modern production facilities. In addition, robotic systems have been actively used to enable patients and people needing assistance to continue their daily lives. This study aims to determine the estimated position of the human's hand so that the mobile manipulator carrying or grasping an object can deliver the object from hand to hand to the designated human. For this, the humans in the environment and their body poses are determined using deep learning on the image taken live from the depth sensor camera on the mobile manipulator. Then, the hand positions of the humans are determined using the determined poses. The distances between the mobile manipulator and each detected person are calculated using the signals received from the depth sensor camera on the mobile manipulator. Thus, the mobile manipulator can create a position vector for each detected human. Using these position vectors, it will be possible for the mobile robot to deliver the object it carries to the desired human more quickly by using different shortest-path algorithms. Moreover, it will be easier for robots that use the proposed approach to assist humans in work environments or patient/elderly care centers.