Immersive Telepresence via a Humanoid Robotic Head Using a VR-Headset with Real-Time Stereoscopic Vision and Binaural Audio

Abstract The rapid evolution of technology has created new opportunities for teleoperation, especially with the emergence of affordable Virtual Reality (VR) systems originally designed for entertainment and gaming. Teleoperation in robotics has a long history, however, its integration with VR for controlling robotic systems is currently gaining significant attention. In this study, a novel real-time platform is developed that integrates teleoperation with a VR system, providing immersive control of a biomimetic robotic head. A 3D model of the robotic head was designed and built to mimic the movements of the operator's head in real-time using the operator’s kinematics. The fabricated robotic head is equipped with a pair of cameras and a binaural microphone to provide vision and audio perception. Additionally, two servomotors are employed to track the motion of the operator’s neck. To control the biomimetic robotic head, a game was developed in Unity to deliver 3D visual and stereophonic audio inputs to the headset, which ensures a smooth viewing and listening experience for the operator. The designed platform is operated through a custom-written C + + program that facilitates continuous tracking of the headset and establishes constant communication with the servomotors. High precision in the positional data and accurate tracking of the head-mounted display (HMD), along with efficient communication with the motors, were achieved. The integration of the VR system and teleoperation developed in this study advances VR technology beyond traditional entertainment-oriented applications and makes it applicable in diverse industrial fields, simultaneously elevating teleoperation in human–robot interactions to new levels of productivity.